Image processing method and device for large model, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method and device for a large model, a storage medium and a program product, and relates to the technical field of big data and image processing, and the method comprises the steps: obtaining a window image of a shooting window of a camera; cutting and removing an invalid area in the window image to obtain an effective window image; adjusting a first exposure parameter of the camera according to the effective window image; controlling a camera to shoot according to the first exposure parameter to obtain at least one to-be-recognized image; and inputting the at least one to-be-identified image into a large model to participate in large model processing. The window image is cut, the exposure parameter is adjusted according to the effective window image obtained through cutting, the image input into the large model is obtained through the exposure parameter, and the problem that under the condition of image overexposure or underexposure, the prediction accuracy degree of the large model can be reduced is solved.

Description

Technical Field

[0001] This disclosure relates to the fields of big data and image processing technology, and in particular to an image processing method, device, storage medium, and program product for large models. Background Technology

[0002] With the development of artificial intelligence, large models have gained the ability to generate results by combining images. However, this ability is affected by the quality of the input images. High-quality images can not only significantly improve the model's prediction, analysis, and forecasting capabilities, but also ensure the accuracy of the results.

[0003] Currently, in some scenarios (such as wearable terminal devices), users cannot check the real-time shooting status of images and cannot determine whether the images are overexposed or underexposed.

[0004] However, the inventors discovered that the related technology has at least the following technical problems: the prediction accuracy of the large model decreases when the image is overexposed or underexposed. Summary of the Invention

[0005] This disclosure provides an image processing method, apparatus, storage medium, and program product for large models to address the problem that the prediction accuracy of large models decreases when images are overexposed or underexposed.

[0006] In a first aspect, embodiments of this disclosure provide an image processing method for a large model, comprising: acquiring a viewport image of a camera's shooting window; cropping and removing invalid regions from the viewport image to obtain a valid viewport image; adjusting a first exposure parameter of the camera based on the valid viewport image; controlling the camera to take a picture according to the first exposure parameter to obtain at least one image to be identified; and inputting the at least one image to be identified into a large model for participation in large model processing.

[0007] Secondly, embodiments of this disclosure provide an image processing device for a large model, comprising: an image acquisition module for acquiring a window image of a camera's shooting window; an image cropping module for cropping and removing invalid regions from the window image to obtain a valid window image; a parameter adjustment module for adjusting a first exposure parameter of the camera according to the valid window image; an image capturing module for controlling the camera to capture images according to the first exposure parameter to obtain at least one image to be recognized; and a model processing module for inputting at least one image to be recognized into a large model for participation in large model processing.

[0008] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing at least one processor to perform the image processing method for large models as described in the first aspect above and various possible designs of the first aspect.

[0009] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image processing method for large models as described in the first aspect and various possible designs of the first aspect.

[0010] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the image processing method for large models as described in the first aspect above and various possible designs of the first aspect.

[0011] The image processing method, device, storage medium, and program product for large models provided in this embodiment acquire a window image, crop the invalid areas in the window image to obtain an effective window image, use the effective window image to determine exposure parameters, use the determined exposure parameters to capture an image to be identified, and input the image to be identified into the large model for subsequent processing. This avoids the influence of invalid areas on exposure, and increases exposure accuracy without requiring the user to observe the exposure or manually adjust the exposure parameters, thereby making the content output by the large model more accurate. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method for large models provided in the embodiments of this application.

[0014] Figure 2 A schematic flowchart illustrating the image processing method for large models provided in an embodiment of this application;

[0015] Figure 3 A schematic diagram of the invalid region provided in the embodiments of this application;

[0016] Figure 4 This is a schematic diagram of the window image and the effective window image provided in the embodiments of this application;

[0017] Figure 5 This is a schematic diagram of labeled image data provided in an embodiment of this application;

[0018] Figure 6 A schematic diagram illustrating the large model processing procedure for at least two images to be identified, provided in an embodiment of this application.

[0019] Figure 7 This is a schematic diagram of dual point of interest region identification provided in an embodiment of this application;

[0020] Figure 8 This is a schematic diagram of the structure of an image processing device for large models provided in an embodiment of this application;

[0021] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] With the continuous advancement of artificial intelligence technology, large models can now generate relevant results from images. However, the effectiveness of this function is highly dependent on the quality of the input images. Specifically, high-quality images can enhance the model's predictive, analytical, and forecasting performance, and strongly guarantee the accuracy of the output results.

[0024] Currently, in some applications (such as wearable smart devices), users are limited by the inability to instantly review the captured image, making it difficult to determine whether the image is overexposed or underexposed. In cases of overexposure or underexposure, details in the image are lost, especially in highlight areas (i.e., white areas), where details are "washed away." These lost details and information are crucial for inference tasks performed by large models. Specifically, overexposure obscures important details in the image (such as texture, edges, and shapes) and causes color information to become blurred or bleached, preventing the model from correctly extracting or distinguishing these features. Therefore, large models encounter difficulties in feature extraction and object recognition, leading to misclassification or complete failure to recognize objects, especially in tasks that heavily rely on visual information. The prediction accuracy of large models decreases in cases of overexposure or underexposure.

[0025] To address the aforementioned technical problems, the inventors propose the following technical concept: by acquiring the view image of the camera's shooting window, cropping the invalid area in the view image to obtain the valid view image, adjusting the camera's exposure parameters based on the valid view image, controlling the camera to take a picture according to the adjusted exposure parameters, obtaining the image to be recognized, and inputting the image to be recognized into a large model for processing.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method for large models provided in the embodiments of this application. For example... Figure 1 In this scenario, the components include: terminal device 101, edge terminal 102, and server 103.

[0027] In the specific implementation process, the terminal device 101 may include headphones, VR (Virtual Reality) devices, smart glasses, etc.

[0028] Edge terminal 102 may include computers, servers, tablets, mobile phones, PDAs (personal digital assistants), and laptops, etc., which can input data.

[0029] Server 103 can be implemented using a single server or a cluster of multiple servers with more powerful processing capabilities and higher security. Where possible, it can also be replaced by a computer or laptop with strong computing power.

[0030] Terminal device 101 can be connected to edge terminal 102, and edge terminal 102 can be connected to server 103. In one possible implementation, terminal device 101 can also be directly connected to server 103. The connection method of the above three devices can be wired or wireless.

[0031] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the image processing method for large models. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented by hardware, software, or a combination of both.

[0032] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0033] Figure 2 This is a flowchart illustrating an image processing method for large models provided in an embodiment of this application. The execution entity of this embodiment may be... Figure 1 The terminal device 101 in the embodiment has a camera, but this embodiment does not impose any special limitations on it. Figure 2 As shown, the method includes:

[0034] S201: Obtain the view image of the camera's shooting window.

[0035] This step may include acquiring the window image using the interface of the shooting window, or it may include acquiring the window image of the shooting window using a preset script or command.

[0036] S202: Crop out invalid areas in the viewport image to obtain a valid viewport image.

[0037] This step may include cropping the preset area from the window image, treating it as an invalid area, and determining the remaining portion of the window image as the valid window image. Alternatively, it may include identifying POI (Point of Interest) regions in the window image, cropping the image outside the POI regions, and obtaining the valid window image.

[0038] S203: Adjust the camera's first exposure parameters based on the valid viewport image.

[0039] This step may include determining the brightness or exposure of the effective viewport image, and adjusting the camera's exposure parameters based on the brightness or exposure. When the brightness or exposure is high, the shutter speed is increased, the aperture is decreased, and the ISO is decreased; when the brightness or exposure is low, the shutter speed is decreased, the aperture is increased, and the ISO is increased. Alternatively, it may include directly inputting the effective viewport image into a pre-trained exposure parameter model to obtain the first exposure parameters output by the model.

[0040] The exposure parameters include aperture, shutter speed, and ISO (sensitivity).

[0041] S204: Control the camera to take a picture according to the first exposure parameters to obtain at least one image to be identified.

[0042] This step may include inputting the first exposure parameters into the camera module and executing a shooting command to obtain at least one image to be recognized. Alternatively, it may include directly writing the first exposure parameters into the shooting command, executing the shooting command, and obtaining the image to be recognized captured by the camera.

[0043] S205: Input at least one image to be recognized into the large model and participate in the large model processing.

[0044] This step may include inputting the image to be recognized into the large model of the terminal device 101 for large model processing, or inputting the image to be recognized into the edge terminal 102 for large model processing. Alternatively, the terminal device 101 may send the image to be recognized to the server 103 via the edge device 102, whereby the server inputs the image into the large model for large model processing.

[0045] As can be seen from the description of the above embodiments, the embodiments of this disclosure obtain a window image, crop the invalid area in the window image to obtain a valid window image, use the valid window image to determine the exposure parameters, use the determined exposure parameters to capture the image to be identified, and input the image to be identified into a large model for subsequent processing. This avoids the influence of invalid areas on exposure, and increases the exposure accuracy without requiring the user to observe the exposure situation or manually adjust the exposure parameters, thereby making the content output by the large model more accurate.

[0046] In one possible implementation, the terminal device has at least one camera in operation.

[0047] For example, a terminal device may have two cameras, one of which is in sleep or off state and the other is in running state, or a terminal device may have four cameras, two of which are in running state.

[0048] Accordingly, in step S202 above, cropping and removing invalid regions from the viewport image to obtain a valid viewport image includes:

[0049] S202A1: Determine invalid areas in the viewport image based on the shooting position of at least one camera in operation.

[0050] In this step, the camera is positioned, for example, beside the user's ear, near the ear, or next to the face.

[0051] The shooting location can be determined by the preset type of the terminal device or it can be preset.

[0052] S202A2: Remove invalid regions from the window image to obtain a valid window image.

[0053] This step may include removing pixels from invalid regions of the viewport image to obtain a valid viewport image; or it may include ignoring invalid regions in subsequent processing.

[0054] As can be seen from the description of the above embodiments, the embodiments of this disclosure remove invalid areas in the shooting window and the invalid areas corresponding to the shooting position in the window image to obtain an effective window image, thereby removing interference factors in the image and ensuring the accuracy of exposure adjustment.

[0055] In one possible implementation, the terminal device has two cameras in operation.

[0056] Accordingly, in step S202A1 above, determining the invalid region in the window image based on the shooting position of at least one camera in operation includes:

[0057] S2A11: Determine invalid areas in the viewport image based on the shooting positions of the two cameras in operation.

[0058] This step is similar to step S202A1 above. For example, when the two cameras are on either side of the user's cheeks, the cheek area in the viewport image is determined as an invalid area.

[0059] The cheek area can be determined using methods such as percentile P95 or PCA (long face, wide face) head models. It is worth noting that the data used to determine the cheek area can be obtained with prior authorization and comply with relevant regulations.

[0060] Figure 3 This is a schematic diagram of an invalid region provided in an embodiment of this application. For example... Figure 3 As shown, the shaded area represents the cheek area (invalid area).

[0061] Figure 4 This is a schematic diagram of the window image and the effective window image provided in the embodiments of this application. Figure 4 As shown, the top image is a viewport image. Two cameras capture images from the left and right sides of the user, respectively. After cropping the portion of the face that is obscured by the cheeks, the final image is obtained. Figure 4 The image below shows the portion of the image occluded by the user's face. Facial occlusion typically occupies a large proportion of the image and lacks extractable features, rendering it ineffective for large models. During subsequent processing such as exposure adjustment and histogram equalization, excessive brightness or darkness in these areas can affect the overall grayscale distribution, potentially leading to unsatisfactory image results after exposure parameter adjustments and grayscale equalization. Therefore, cropping to remove these invalid areas can improve image quality in subsequent processing.

[0062] As can be seen from the description of the above embodiments, the embodiments of this disclosure determine the invalid area in the window image based on the shooting positions of the two cameras in operation, so that the content of the cropped part can be supplemented by the cameras on both sides. Therefore, the content information in the overall shooting field of view will not be affected, and the influence of the invalid area on the subsequent exposure adjustment is also eliminated.

[0063] In one possible implementation, step S202 above, which involves cropping and removing invalid regions from the viewport image to obtain a valid viewport image, includes:

[0064] S202B1: Input the viewport image into the interest point determination model to obtain the interest point region output by the interest point determination model.

[0065] In this step, the interest point determination model can be a multi-layer language model pre-trained by the staff. The interest point region can be delineated using pixel coordinates.

[0066] S202B2: Crop the area outside the point of interest region in the viewport image to obtain a valid viewport image.

[0067] This step is similar to step S202A2 above, where areas outside the point of interest are treated as invalid areas and cropped to obtain a valid viewport image.

[0068] As described in the above embodiments, this embodiment of the present disclosure inputs a viewport image into an interest point determination model to obtain an interest point region output by the interest point determination model, and then crops the image outside the interest point region to obtain an effective viewport image, thereby eliminating the influence of the area outside the interest point region on subsequent exposure adjustments.

[0069] In one possible implementation, the interest point determination model is trained using the following method:

[0070] S220: Obtain the labeled image data and the corresponding language description of the labeled image data.

[0071] The labeled image data can be obtained by staff using a segmentation model to divide the labeled image data and then manually labeling the divided image data, or it can be obtained by staff manually selecting and delineating in the image data.

[0072] Figure 5 This is a schematic diagram of labeled image data provided in an embodiment of this application. For example... Figure 5 As shown, there are three types of labeled image data: A, B, and C. Image A is a manually labeled image obtained after segmentation using a segmentation model; image B is a manually labeled image obtained by selecting bounding boxes; and image C is an image in which objects were manually delineated and painted over by staff. Figure 5 The labeled image is obtained from the shaded area. The description can be entered by the staff or collected along with the image data.

[0073] S221: Input the labeled image data and corresponding language descriptions into the large model to be trained, so that the large model to be trained can perform comparative learning and obtain the interest point determination model.

[0074] In this step, the large model to be trained can also be a multi-layer language model. This involves inputting labeled image data and corresponding language descriptions into the large model, which then uses an image encoder to convert the image data into image vectors and a text encoder to convert the language descriptions into language vectors. The similarity between the image vectors and the language vectors is calculated, and this similarity is input into a loss function to obtain the loss value. The parameters of the image encoder and text encoder are updated using the loss value. This step is repeated until the loss value is less than a preset value, resulting in an interest point determination model. The interest point determination model can further improve its ability to identify user POI regions through fine-tuning, reinforcement learning, Few-Shot learning (constructing data), and incorporating prior knowledge (such as depth information, distinctive object features, and color). The contrastive learning process can also include pre-training the model on a specific task to improve the accuracy of inferences through model optimization. Pre-training methods include CLIP (Contrastive Language-Image Pre-training), GLIP (Grounded Language-Image Pre-training), and SEEM (SegmentEverything Everywhere All at Once) pre-training (contrastive language-image pre-training). Reinforcement learning can also be used to guide model optimization by collecting human feedback. During contrastive learning, researchers can score the recognition results generated by the model, and the model learns to generate more accurate outputs based on these scores.

[0075] As can be seen from the description of the above embodiments, the embodiments of this disclosure input labeled image data and the corresponding language descriptions into a large model to be trained, and use a contrastive learning method to train the large model to be trained to obtain an interest point determination model, which is convenient for subsequent use of the interest point determination model to determine interest points in the image.

[0076] In one possible implementation, step S202 above, which involves cropping and removing invalid regions from the viewport image to obtain a valid viewport image, includes:

[0077] S202C1: Input the window image and the preset device type into the interest point determination model to obtain the interest point region output by the interest point determination model.

[0078] This step may include fusing the window image and the device type itself and inputting it into the interest point determination model to obtain the interest point region output by the interest point determination model.

[0079] The training process for the interest point determination model in this step can be similar to the training method in the above embodiments, with the corresponding device type added as input during training, or the training dataset being associated with the camera's location / device type. This will not be elaborated further here. For AR / VR devices and headphones, paired image acquisition is performed by calculating the overlap between the different effective ranges of the camera and the user's everyday field of view.

[0080] S202C2: Crops the area outside the point of interest region in the viewport image to obtain a valid viewport image.

[0081] This step is similar to step S202B2 above, and will not be repeated here.

[0082] As can be seen from the description of the above embodiments, the embodiments of this disclosure, by also inputting the type of the device itself into the point of interest determination model, enable the model to more accurately determine the points of interest in the image by combining the type of the device.

[0083] In one possible implementation, step S203 above, adjusting the first exposure parameters of the camera based on the effective viewport image, includes:

[0084] S2031: Input the effective window image into the exposure parameter determination model to obtain the first exposure parameter output by the exposure parameter determination model.

[0085] In this step, the exposure parameter determination model can be any of the following: a feedforward neural network model, a feedback neural network model, etc.

[0086] As can be seen from the description of the above embodiments, the embodiments of this disclosure obtain the first exposure parameter by inputting the effective window image into the exposure parameter determination model, thereby achieving accurate exposure parameter determination, which facilitates the subsequent obtaining of an accurately exposed image and increases the accuracy of the output of the large model.

[0087] In one possible implementation, after obtaining the window image of the camera's shooting window in step S201 above, the method further includes:

[0088] S230: Receive semantic data input by the user.

[0089] This step includes using the microphone built into the terminal device to receive semantic data input by the user.

[0090] Semantic data includes phrases such as "How is this flower growing?", "What is the model number of this product?", and "What is this?".

[0091] S231: Determine the second exposure parameters based on the window image and the semantic data.

[0092] This step includes directly inputting the window image and semantic data into the large model to obtain exposure parameters; it may also include cropping and removing invalid areas in the window image based on the window image and semantic data to obtain an effective window image, and determining the second exposure parameters based on the effective window image.

[0093] S232: Control the camera to take a picture according to the second exposure parameters to obtain at least one image to be identified.

[0094] This step is similar to step S204 above, and will not be repeated here.

[0095] S233: Input at least one image to be identified and its semantic data into the large model to participate in the large model processing.

[0096] This step is similar to step S205 above, and will not be repeated here.

[0097] As can be seen from the description of the above embodiments, the embodiments of this disclosure combine window images and semantic data to adjust exposure parameters, thereby making the exposure parameters more accurate and improving the processing effect of large models.

[0098] In one possible implementation, after step S204 above, where the camera is controlled to capture an image according to the first exposure parameters to obtain at least one image to be identified, the method further includes:

[0099] S2041: Cropping and removing invalid regions from the image to be recognized to obtain a valid image to be recognized.

[0100] This step is similar to step S202 above, and will not be repeated here.

[0101] S2042: Perform histogram equalization on the valid image to be identified to obtain the equalized image to be identified.

[0102] This step includes calculating the grayscale histogram of the effective image to be identified, which is to count the number of times each grayscale level appears; calculating the cumulative distribution function; performing a linear transformation using the cumulative distribution function to obtain new grayscale values; and applying the new grayscale values ​​to the image to obtain the equalized image.

[0103] The grayscale histogram is a bar chart representing the frequency of each grayscale value (0 to 255) in the image. The horizontal axis represents the grayscale level (0 to 255), and the vertical axis represents the number of pixels at the corresponding grayscale level. The cumulative distribution function is obtained by summing the histograms; it represents the total cumulative frequency from zero to a certain grayscale value. The linear transformation uses the cumulative distribution function to map the original grayscale values ​​to new grayscale values.

[0104] S2043: Input the image to be identified into the large model and participate in the large model processing.

[0105] This step is similar to step S205 above. The input image to be recognized for equalization can be one or multiple images.

[0106] As can be seen from the description of the above embodiments, the embodiments of this disclosure obtain a valid image to be identified by cropping the invalid region in the image to be identified, performing histogram equalization to obtain a balanced image to be identified, and inputting the balanced image to be identified into a large model to participate in the large model processing, thereby flattening the frequency distribution of gray values ​​in the histogram, making the distribution of gray values ​​more uniform, thereby improving the contrast and detail of the image.

[0107] In one possible implementation, after step S203 above, which adjusts the first exposure parameters of the camera based on the effective viewport image, the method further includes:

[0108] S240: Determine whether the exposure is overexposed or underexposed.

[0109] This step may include re-acquiring the viewport image, determining its histogram, and identifying underexposure or overexposure if the pixels in the histogram are concentrated near the left or right side. It may also include taking a picture to obtain the image to be identified, determining its histogram, and using the histogram to determine if the image is underexposed or overexposed. Alternatively, it may include inputting the image to be identified into a large model, which will then determine if it is overexposed or underexposed.

[0110] S240: If the image is overexposed or underexposed, output a prompt message to adjust the shooting position to optimize the exposure.

[0111] In this step, the prompts can be pre-set, with corresponding preset prompts for overexposure or underexposure. The prompts can also be from the output of a large model.

[0112] The prompts may include messages such as, "The image being captured may be overexposed; you may need to adjust your position."

[0113] As can be seen from the description of the above embodiments, the embodiments of this disclosure, after optimizing the exposure parameters, determine whether the image is overexposed or underexposed again, and output a prompt message to adjust the shooting position to optimize the exposure in the case of overexposure or underexposure, thereby prompting the user to adjust the shooting position, allowing the user to adjust their posture or position in order to obtain a better shooting angle and lighting conditions, and increase the usability of subsequent images.

[0114] In one possible implementation, after obtaining the window image of the camera's shooting window in step S201 above, the method further includes:

[0115] S250: Obtain hardware resource status.

[0116] In this step, the hardware resource status includes one or more of the following: remaining battery power, available bandwidth, and the amount of idle computing resources.

[0117] Accordingly, in step S204 above, the camera is controlled to take a picture according to the first exposure parameters to obtain at least one image to be identified, including:

[0118] S2041: If the hardware resource status does not meet the preset requirements, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized.

[0119] This step may include inputting the status of each hardware resource into a pre-written judgment program to determine whether the hardware resource status meets preset requirements. If it does not meet the requirements, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized. Alternatively, it may include inputting the status of each hardware resource into a pre-set hardware resource score calculation formula to obtain a hardware resource score. If the hardware resource score is less than a preset resource score threshold, the hardware resource status is determined to not meet the preset requirements; otherwise, the hardware resource status is determined to meet the preset requirements.

[0120] Among them, the preset requirements include, for example, if the remaining battery power is greater than or equal to a preset remaining battery power threshold and the number of idle computing resources is greater than or equal to a preset number of computing resources threshold, then the preset requirements are determined to be met; or if the available bandwidth is greater than or equal to a preset bandwidth threshold, then the preset requirements are determined to be met; otherwise, the preset requirements are determined not to be met.

[0121] S2042: If the hardware resource status meets the preset requirements, the camera is controlled to take pictures according to the first exposure parameters to obtain at least two images to be identified.

[0122] In this step, whether the preset requirements are met can be achieved by adopting the scheme of step S2041 above, capturing at least two images to be identified, which may include executing the shooting program at least twice using the first exposure parameter to obtain at least two images to be identified, or executing the video recording program using the first exposure parameter to obtain a video, the video consisting of at least two images to be identified.

[0123] As can be seen from the description of the above embodiments, the embodiments of this disclosure determine whether the hardware resources meet the preset requirements and select to take one or at least two pictures, so as to make the number of image processing conform to the hardware resource status, increase the processing speed of large models, and improve the effect of automatic exposure adjustment under the premise of minimizing computing power / computing power allowance, and ensure that the final generated image has high quality.

[0124] In one possible implementation, if at least two images to be recognized are obtained, different equalization methods can be applied to each image during the equalization process, but all of them must meet the established criteria. Using at least two (a set of) images to be recognized helps large models obtain more accurate inference results.

[0125] In one possible implementation, the hardware resource status includes at least one of power, bandwidth, and computing resources.

[0126] Among these, power consumption, bandwidth, and computing resources can be either the terminal device itself or the edge terminals connected to the terminal device.

[0127] Accordingly, if the hardware resource status does not meet the preset requirements in step S2041, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized, including: S2041A, S2041B or S2041C.

[0128] S2041A: If the battery level is less than the preset battery threshold, the camera will be controlled to take a picture according to the first exposure parameters to obtain an image to be recognized.

[0129] In this step, the preset power threshold can be pre-set by the staff based on experimental or empirical parameters.

[0130] For example, when the battery level is less than 20%, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be identified; as another example, when the battery level is less than 10%, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be identified.

[0131] S2041B: If the bandwidth is less than the preset bandwidth threshold, the camera is controlled to take a picture according to the first exposure parameter to obtain an image to be recognized.

[0132] In this step, the preset bandwidth threshold can be pre-set by the staff based on experimental or empirical parameters.

[0133] For example, if the bandwidth is less than 20Mbps, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be identified. As another example, if the bandwidth is less than 15Mbps, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be identified. Yet another example, if the bandwidth is less than 30Mbps, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be identified.

[0134] S2041C: If the computing resources are less than the preset computing resource threshold, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized.

[0135] In this step, the preset computing resource threshold can be pre-set by staff based on experimental or empirical parameters.

[0136] For example, if the current computing resources are 1 core remaining and the preset computing resource threshold is 3 cores, then the camera will be controlled according to the first exposure parameters to capture an image to be recognized. As another example, if the current computing resources are 2 cores remaining and the preset computing resource threshold is 4 cores, then the camera will be controlled according to the first exposure parameters to capture an image to be recognized. And as yet another example, if the current computing resources are 4 cores remaining and the preset computing resource threshold is 5 cores, then the camera will be controlled according to the first exposure parameters to capture an image to be recognized.

[0137] As can be seen from the description of the above embodiments, the embodiments of this disclosure reduce the user's waiting time by checking the status of power, bandwidth, and computing resources, and using fewer images to be identified for subsequent processing when the preset standards are not met.

[0138] Figure 6 This is a schematic diagram illustrating the large-model processing procedure for at least two images to be identified, provided in an embodiment of this application. (See attached diagram.) Figure 6 As shown, at least two images can be combined to form a time series image. The large model determines the current point of interest location by recognizing the position and gesture of the hand in the image to be recognized. Figure 6 In the image, a series of images represents the user's hand circling at item B, thus item B is the point of interest. During the recognition process, other actions can also be identified, such as pointing and clicking. It can also recognize changes in the object being pointed at.

[0139] Figure 7 This is a schematic diagram illustrating dual-point-of-interest region identification provided in an embodiment of this application. Figure 7 As shown, in a scene, if a user points to two objects A and B with both hands, the regions where objects A and B are identified are both designated as points of interest.

[0140] Figure 8 This is a schematic diagram of the structure of an image processing device for large models provided in an embodiment of this application. Figure 8 As shown, the image processing device 800 for large models includes:

[0141] Image acquisition module 801 is used to acquire the window image of the camera's shooting window;

[0142] Image cropping module 802 is used to crop and remove invalid areas in the view image to obtain a valid view image;

[0143] The parameter adjustment module 803 is used to adjust the first exposure parameters of the camera based on the effective view window image;

[0144] The image capture module 804 is used to control the camera to capture images according to the first exposure parameters, so as to obtain at least one image to be recognized.

[0145] The model processing module 805 is used to input at least one image to be recognized into the large model for large model processing.

[0146] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0147] In one possible implementation, the terminal device has at least one camera in operation;

[0148] Accordingly, the image cropping module 802 is specifically used to determine the invalid region in the window image based on the shooting position of at least one camera in operation; remove the invalid region in the window image to obtain a valid window image.

[0149] In one possible implementation, the terminal device has two cameras in operation; the image cropping module 802 is specifically used to determine the invalid area in the window image based on the shooting positions of the two cameras in operation.

[0150] In one possible implementation, the image cropping module 802 is specifically used to input the window image into the interest point determination model to obtain the interest point region output by the interest point determination model; and to crop the area outside the interest point region in the window image to obtain a valid window image.

[0151] In one possible implementation, the interest point determination model is trained by: acquiring labeled image data and corresponding language descriptions; inputting the labeled image data and corresponding language descriptions into a large model to be trained, so that the large model to be trained can perform comparative learning to obtain the interest point determination model.

[0152] In one possible implementation, the image cropping module 802 is specifically used to input the window image and the preset device type into the interest point determination model to obtain the interest point region output by the interest point determination model; and to crop the area outside the interest point region in the window image to obtain a valid window image.

[0153] In one possible implementation, the parameter adjustment module 803 is specifically used to input the effective window image into the exposure parameter determination model to obtain the first exposure parameter output by the exposure parameter determination model.

[0154] In one possible implementation, the image processing device 800 for large models also includes a comprehensive processing module 806.

[0155] The integrated processing module 806 is specifically used to receive semantic data input by the user; determine the second exposure parameter based on the window image and the semantic data; control the camera to take a picture according to the second exposure parameter to obtain at least one image to be recognized; and input the at least one image to be recognized and the semantic data into the large model to participate in the large model processing.

[0156] In one possible implementation, the large-scale image processing device 800 also includes an image equalization module 807.

[0157] The image equalization module 807 is used to crop and remove invalid regions in the image to be recognized to obtain a valid image to be recognized; to perform histogram equalization on the valid image to be recognized to obtain an equalized image to be recognized; and to input the equalized image to be recognized into the large model to participate in the large model processing.

[0158] In one possible implementation, the image processing device 800 for large models also includes an exposure prompting module 808.

[0159] The exposure prompt module 808 is specifically used to determine whether the exposure is overexposed or underexposed; if it is overexposed or underexposed, it outputs a prompt message to adjust the shooting position to optimize the exposure.

[0160] In one possible implementation, the large-model image processing device 800 also includes a resource acquisition module 809.

[0161] The resource acquisition module 809 is used to acquire the hardware resource status; correspondingly, the image capturing module 804 is specifically used to control the camera to capture an image according to the first exposure parameters if the hardware resource status does not meet the preset requirements, thereby obtaining an image to be recognized; if the hardware resource status meets the preset requirements, the camera is controlled to capture an image according to the first exposure parameters, thereby obtaining at least two images to be recognized.

[0162] In one possible implementation, the hardware resource status includes at least one of power, bandwidth, and computing resources; correspondingly, the image capturing module 804 is specifically used to control the camera to capture an image according to the first exposure parameters if the power is less than a preset power threshold, thereby obtaining an image to be recognized; or, if the bandwidth is less than a preset bandwidth threshold, control the camera to capture an image according to the first exposure parameters, thereby obtaining an image to be recognized; or, if the computing resources are less than a preset computing resource threshold, control the camera to capture an image according to the first exposure parameters, thereby obtaining an image to be recognized.

[0163] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0164] To implement the above embodiments, this disclosure also provides an electronic device.

[0165] refer to Figure 9 The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device such as VR glasses, AR glasses, or headphones with a camera. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0166] like Figure 9 As shown, electronic device 900 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from storage device 908 into random access memory (RAM) 903. RAM 903 also stores various programs and data required for the operation of electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via bus 904 (or other lines, circuit boards). Input / output (I / O) interface 905 is also connected to bus 904.

[0167] Typically, the following devices can be connected to I / O interface 905: input devices 906, such as cameras and microphones; output devices 907, such as liquid crystal displays (LCDs), speakers, and vibrators; and communication devices 908. Communication device 908 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0168] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 908, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0169] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the technical solution of the image processing method for large models in any of the above embodiments. The implementation principle and beneficial effects are similar to those of the image processing method for large models. Please refer to the implementation principle and beneficial effects of the image processing method for large models, which will not be repeated here.

[0170] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0171] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the image processing method for large models in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the image processing method for large models, and can be found in the implementation principle and beneficial effects of the image processing method for large models, which will not be repeated here.

[0172] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0173] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0174] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0177] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0178] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0179] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0180] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0181] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method for large models, characterized in that, Applied to terminal devices with cameras, including: Acquire the view image of the camera's capture window; The invalid regions in the viewport image are cropped and removed to obtain a valid viewport image; Adjust the first exposure parameters of the camera based on the effective viewport image; The camera is controlled to take a picture according to the first exposure parameters to obtain at least one image to be identified; The at least one image to be identified is input into the large model for processing.

2. The method according to claim 1, characterized in that, The terminal device has at least one camera in operation; Accordingly, the cropping process to remove invalid regions from the viewport image to obtain a valid viewport image includes: Based on the shooting position of the at least one camera in operation, determine the invalid area in the window image; Remove invalid regions from the viewport image to obtain a valid viewport image.

3. The method according to claim 2, characterized in that, The terminal device has two cameras in operation. Accordingly, determining the invalid region in the viewport image based on the shooting position of the at least one operating camera includes: Based on the shooting positions of the two cameras in operation, the invalid areas in the window image are determined.

4. The method according to claim 1, characterized in that, The process of cropping and removing invalid regions from the viewport image to obtain a valid viewport image includes: The window image is input into the interest point determination model to obtain the interest point region output by the interest point determination model. The region outside the point of interest area in the viewport image is cropped to obtain a valid viewport image.

5. The method according to claim 4, characterized in that, The interest point determination model was trained using the following method: Obtain labeled image data and the corresponding language descriptions for the labeled image data; The labeled image data and corresponding language descriptions are input into the large model to be trained, so that the large model to be trained can perform comparative learning to obtain the interest point determination model.

6. The method according to claim 1, characterized in that, The process of cropping and removing invalid regions from the viewport image to obtain a valid viewport image includes: The window image and the preset device type are input into the interest point determination model to obtain the interest point region output by the interest point determination model; The region outside the point of interest area in the viewport image is cropped to obtain a valid viewport image.

7. The method according to claim 1, characterized in that, The step of adjusting the first exposure parameters of the camera based on the effective viewport image includes: The effective window image is input into the exposure parameter determination model to obtain the first exposure parameter output by the exposure parameter determination model.

8. The method according to claim 1, characterized in that, After acquiring the image of the camera's shooting window, the method further includes: Receive semantic data input from the user; The second exposure parameter is determined based on the window image and the semantic data. The camera is controlled to take a picture according to the second exposure parameters to obtain at least one image to be identified; The at least one image to be identified and the semantic data are input into the large model for processing.

9. The method according to any one of claims 1 to 8, characterized in that, After controlling the camera to take a picture according to the first exposure parameters to obtain at least one image to be identified, the method further includes: The invalid regions in the image to be identified are cropped and removed to obtain the valid image to be identified; Histogram equalization is performed on the effective image to be identified to obtain an equalized image to be identified; The image to be identified and balanced is input into the large model for processing.

10. The method according to any one of claims 1 to 8, characterized in that, After adjusting the first exposure parameters of the camera based on the effective viewport image, the method further includes: Determine if the exposure is overexposed or underexposed; If the image is overexposed or underexposed, a prompt message will be output suggesting adjusting the shooting position to optimize the exposure.

11. The method according to any one of claims 1 to 8, characterized in that, After acquiring the image of the camera's shooting window, the method further includes: Get the status of hardware resources; Accordingly, controlling the camera to take a picture according to the first exposure parameters to obtain at least one image to be identified includes: If the hardware resource status does not meet the preset requirements, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized. If the hardware resource status meets the preset requirements, the camera is controlled to take pictures according to the first exposure parameters to obtain at least two images to be identified.

12. The method according to claim 11, characterized in that, The hardware resource status includes at least one of power, bandwidth, and computing resources; Accordingly, if the hardware resource status does not meet the preset requirements, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized, including: If the battery level is less than a preset battery threshold, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized; or, If the bandwidth is less than a preset bandwidth threshold, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized; or, If the computing resources are less than a preset computing resource threshold, the camera is controlled to take a picture according to the first exposure parameters to obtain an image to be recognized.

13. An image processing device for large models, characterized in that, include: The image acquisition module is used to acquire the image of the camera's shooting window; The image cropping module is used to crop and remove invalid areas from the view image to obtain a valid view image; The parameter adjustment module is used to adjust the first exposure parameters of the camera according to the effective view window image; An image capture module is used to control the camera to capture images according to the first exposure parameters, so as to obtain at least one image to be recognized. The model processing module is used to input the at least one image to be recognized into the large model for large model processing.

14. An electronic device, characterized in that, include: Camera, processor, and memory; The memory stores computer-executed instructions; The processor controls the camera and executes computer execution instructions stored in the memory, causing the processor to perform the image processing method for large models as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image processing method for large models as described in any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for image processing of large models as described in any one of claims 1 to 12.

Images