Image processing method, model training method, device, computer equipment and medium

Abstract

The present disclosure provides an image processing method, including: obtaining an image to be processed; obtaining an optimization model; inputting the image to be processed into the input layer of the optimization model, so that the output layer of the optimization model outputs: multiple detection frame information, multiple confidence information for the multiple detection frame information, and multiple saliency information for the multiple detection frame information; and, based on the multiple detection frame information, the multiple confidence information and the multiple saliency information, determining the key target in the image to be processed. The present disclosure also provides an image processing device, a model training method and device, a computer device and a computer-readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to an image processing method and device, a model training method and device, a computer device and a medium. Background Art

[0002] In the scenario of target detection on images, various target optimization models based on deep learning are usually used to process the images to be processed input by the user to obtain confidence information of multiple regions in the images to be processed. The confidence information is used as the basis for sorting multiple regions to determine the key targets. The problem with this method is that the confidence information is only used to measure the degree to which a region can represent the object target, without considering the prominence of the object target represented by the region in the image to be processed, while users in real life often pay attention to objects with higher prominence. As a result, in some cases, the key targets determined are not the targets that the user really pays attention to. Summary of the invention

[0003] In view of this, the present disclosure provides an image processing method and apparatus, a model training method and apparatus, a computer device and a medium.

[0004] One aspect of the present disclosure provides an image processing method, comprising: obtaining an image to be processed on the one hand, and obtaining an optimization model on the other hand. The image to be processed is input into the input layer of the optimization model, so that the output layer of the optimization model outputs: multiple detection frame information, multiple confidence information for the multiple detection frame information, and multiple saliency information for the multiple detection frame information. Then, based on the multiple detection frame information, multiple confidence information, and multiple saliency information, a key target in the image to be processed is determined.

[0005] According to an embodiment of the present disclosure, each of the above-mentioned multiple confidence information includes a confidence score, and each of the above-mentioned multiple saliency information includes a saliency score. The above-mentioned determination of the key target in the image to be processed based on multiple detection frame information, multiple confidence information and multiple saliency information includes: based on multiple confidence information, filtering the detection frame information with a confidence score higher than a predetermined threshold from the multiple detection frame information to obtain the detection frame information that has undergone the first screening. Then, based on the multiple saliency information, sorting the detection frame information that has undergone the first screening in order of the saliency score from high to low to obtain the sorting result. Then, filtering the first predetermined number of detection frame information from the sorting result to obtain the detection frame information that has undergone the second screening. Thereby, based on the detection frame information that has undergone the second screening, the key target in the image to be processed is determined.

[0006] According to an embodiment of the present disclosure, the method further includes: after determining the key target in the image to be processed, based on the detection frame information of the key target, segmenting a sub-image containing the key target from the image to be processed, obtaining search results about the sub-image, and then outputting the search results.

[0007] Another aspect of the present disclosure provides a model training method, including: obtaining multiple sample images, marking multiple regions in each of the multiple sample images, and adding a label to any region of the multiple regions. The label includes a first label and a second label, the first label is used to characterize whether any region includes an object target, and the second label is used to characterize whether the object target contained in any region is significant. Then, an initial model is obtained. The initial model is trained using the multiple sample images with labels obtained above to obtain an optimized model, which is used to determine the key targets in the image to be processed.

[0008] According to an embodiment of the present disclosure, the above-mentioned training of the initial model using labeled sample images to obtain an optimized model includes: inputting any sample image from a plurality of sample images into the input layer of the initial model, so that the output layer of the initial model outputs: multiple predicted detection box information about any sample image, multiple predicted confidence information for the multiple predicted detection box information, and multiple predicted significance information for the multiple predicted detection box information. Then, based on the difference between the multiple predicted detection box information and the multiple regions marked in any sample image, the difference between the multiple predicted confidence information and the first label of any sample image, and the difference between the multiple predicted significance information and the second label of any sample image, determine whether the loss function of the initial model has achieved convergence. If not, adjust the parameters of the initial model and repeat the above-mentioned training process. If yes, use the initial model as the above-mentioned optimized model.

[0009] According to an embodiment of the present disclosure, the output layer of the initial model includes: a first output layer, a second output layer and a third output layer. The first output layer is used to output predicted detection box information, the second output layer is used to output predicted confidence information, and the third output layer is used to output saliency information.

[0010] Another aspect of the present disclosure provides an image processing device, comprising: a first acquisition module, a second acquisition module, a detection module, and a determination module. The first acquisition module is used to acquire an image to be processed. The second acquisition module is used to acquire an optimization model. The detection module is used to input the image to be processed into an input layer of the optimization model so that the output layer of the optimization model outputs: multiple detection frame information, multiple confidence information for the multiple detection frame information, and multiple saliency information for the multiple detection frame information. The determination module is used to determine key targets in the image to be processed based on the multiple detection frame information, the multiple confidence information, and the multiple saliency information.

[0011] Another aspect of the present disclosure provides a model training device, including: a third acquisition module, a region annotation module, a label adding module, a fourth acquisition module, and a training module. The third acquisition module is used to acquire multiple sample images. The region annotation module is used to annotate multiple regions in each of the multiple sample images. The label adding module is used to add a label to any region of the multiple regions. Among them, the label includes a first label and a second label, the first label is used to characterize whether any region includes an object target, and the second label is used to characterize whether the object target contained in any region is significant. The fourth acquisition module is used to acquire an initial model. The training module is used to train the initial model using multiple sample images with labels to obtain an optimized model, and the optimized model is used to determine the key targets in the image to be processed.

[0012] Another aspect of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0013] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0014] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the method described above when executed.

[0015] According to the embodiments of the present disclosure, when the image to be processed is processed using the optimization model, not only the detection frame information and confidence information for multiple regions can be output, but also the saliency information for multiple regions can be output. When determining the key targets in the image, not only the proximity between the image features and the object targets is considered, but also the saliency of each object target is considered, so that the key targets that meet the user's attention and expectations can be determined more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1A The exemplary system architecture of the application image processing method and device and the model training method and device according to the embodiments of the present disclosure is schematically shown;

[0018] Figure 1B The application scenarios of the image processing method and device and the model training method and device according to the embodiments of the present disclosure are schematically shown;

[0019] Figure 2 An example schematic diagram of an image to be processed according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 3A The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown;

[0021] Figure 3B A flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown;

[0022] Figure 4A An example schematic diagram of an optimization model according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 4B An example schematic diagram of an image search process according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 5 A flowchart of a model training method according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 6 A block diagram schematically shows an image processing device according to an embodiment of the present disclosure;

[0026] Figure 7 A block diagram schematically shows a model training device according to an embodiment of the present disclosure; and

[0027] Figure 8 The block diagram of a computer device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0031] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] One aspect of the present disclosure provides an image processing method and apparatus, a model training method and apparatus, a computer device and a medium. Among them, the image processing method may include: an image acquisition process, a model acquisition process, a detection process and a key target determination process. On the one hand, in the image acquisition process, an image to be processed is acquired. On the other hand, in the model acquisition process, an optimization model is acquired. Then a detection process is performed, and the image to be processed is input into the input layer of the optimization model so that the output layer of the optimization model outputs: multiple detection box information, multiple confidence information for multiple detection box information, and multiple saliency information for multiple detection box information. Then a key target determination process is performed, and based on the above-mentioned multiple detection box information, multiple confidence information and multiple saliency information, the key targets in the image to be processed are determined.

[0033] Figure 1A The exemplary system architecture 100 to which the image processing method and apparatus and the model training method and apparatus according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1A What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1AAs shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and wireless communication links, etc.

[0035] The terminal devices 101, 102, 103 may be various electronic devices, and may have the same or different computing capabilities, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. Various client applications may be installed on the terminal devices 101, 102, 103, such as image processing applications, picture search applications, etc. (only for example).

[0036] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The server 105 can provide various background support services, such as a background management server that provides image processing results, retrieval models, etc. to the terminal devices 101, 102, 103 (only as an example). The background management server can analyze and process the received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request, etc.) to the terminal device.

[0037] Figure 1B The application scenarios of the image processing method and device and the model training method and device according to the embodiments of the present disclosure are schematically shown. Figure 1B As shown, the user searches for key objects (such as Figure 1B The key target (eg, a car) is marked with a dotted box), and the search results about the key target are obtained through the interaction between the terminal device 101 and the server 105.

[0038] It should be noted that the model training method and / or image processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the model training device and / or image processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The model training method and / or image processing method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, and 103. Accordingly, the model training device and / or image processing device provided in the embodiment of the present disclosure can be set in the terminal devices 101, 102, and 103. Alternatively, the model training method and / or image processing method provided in the embodiment of the present disclosure can also be executed by other servers or server clusters that can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the model training device and / or image processing device provided in the embodiment of the present disclosure can also be set in other servers or server clusters that can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0039] It should be understood that Figure 1A to Figure 1B The number and type of terminal devices, networks and servers in the embodiment are only for illustration purposes. According to actual needs, any number and type of terminal devices, networks and servers may be provided.

[0040] In the scenario of target detection on images, various target optimization models based on deep learning are usually used to process the images to be processed input by the user to obtain confidence information of multiple regions in the images to be processed. The confidence information is used as the basis for sorting multiple regions to determine the key targets. The problem with this method is that the confidence information is only used to measure the degree to which a region can represent the object target, without considering the prominence of the object target represented by the region in the image to be processed, while users in real life often pay attention to objects with higher prominence. As a result, in some cases, the key targets determined are not the targets that the user really pays attention to.

[0041] Figure 2 An example schematic diagram of an image to be processed according to an embodiment of the present disclosure is schematically shown.

[0042] like Figure 2As shown in the figure, after being processed by the common target optimization model, four object targets are marked in the image to be processed with detection frames 1, 2, 3, and 4. The object target represented by detection frame 1 is a flower, the object target represented by detection frame 2 is a black wallet, the object target represented by detection frame 3 is a red card holder, and the object target represented by detection frame 4 is a red leather bag. Among them, the confidence score of detection frame 3 is the highest. Based on the above method of determining key targets based on confidence, the key target determined is the red card holder. However, the user actually hopes to treat the red leather bag in the image as a key target for subsequent processing. Therefore, the detection process failed to meet user expectations.

[0043] According to an embodiment of the present disclosure, an image processing method is provided that can detect key targets in an image more accurately. The method is described below through a legend. It should be noted that the sequence number of each operation in the following method is only used as a representation of the operation for the purpose of description, and should not be regarded as indicating the execution order of the various operations. Unless explicitly stated, the method does not need to be executed in the order shown.

[0044] Figure 3A The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown.

[0045] like Figure 3A As shown, the image processing method may include the following operations S301 to S304.

[0046] In operation S301, an image to be processed is acquired.

[0047] In operation S302, an optimization model is obtained.

[0048] The execution order of operation S301 and operation S302 is not limited. In this example, operation S301 may be executed first and then operation S302. In other examples, operation S302 may be executed first and then operation S301, or both may be executed simultaneously. The optimized model obtained in operation S302 is a trained model.

[0049] Next, in operation S303, the image to be processed is input into the input layer of the optimization model, so that the output layer of the optimization model outputs: multiple detection box information, multiple confidence information for the multiple detection box information, and multiple saliency information for the multiple detection box information.

[0050] Among them, the multiple detection box information output by the output layer of the optimization model represents the location information of multiple areas containing object targets detected by the optimization model in the image to be processed, the multiple confidence information represents the confidence level of each of the above-mentioned multiple areas, and the multiple saliency information represents the saliency level of each of the above-mentioned multiple areas.

[0051] Next, in operation S304, based on the above-mentioned multiple detection box information, multiple confidence information and multiple saliency information, key targets in the image to be processed are determined.

[0052] It should be noted that Figure 3A The image processing method shown can be applied to the client or the server. When applied to the client, the image to be processed can be received from the user, and the optimization model can be obtained from the server or from the client. When applied to the server, the image to be processed can be received from the client, and the optimization model can be obtained from the server or from other servers or server clusters used to train the optimization model.

[0053] Those skilled in the art can understand that when the image processing method according to the embodiment of the present disclosure uses the optimization model to process the image to be processed, it can not only output the detection frame information and confidence information for multiple regions, but also output the saliency information for multiple regions. When determining the key targets in the image, not only the proximity between the image features and the object targets is considered, but also the saliency of each object target is considered, so that the key targets that meet the user's attention and expectations can be determined more accurately.

[0054] According to an embodiment of the present disclosure, each of the above-mentioned multiple confidence information may include a confidence score, and each of the above-mentioned multiple saliency information may include a saliency score. The above-mentioned process of determining the key target in the image to be processed based on multiple detection frame information, multiple confidence information and multiple saliency information may include: based on multiple confidence information, filtering the detection frame information with a confidence score higher than a predetermined threshold from the multiple detection frame information to obtain the detection frame information that has undergone a first screening. Then, based on the multiple saliency information, sorting the detection frame information that has undergone the first screening in order from high to low according to the saliency score to obtain a first sorting result. Then, filtering the first predetermined number of detection frame information from the first sorting result to obtain the detection frame information that has undergone a second screening, and the detection frame information that has undergone the second screening corresponds to the key target in the image to be processed. Thus, based on the detection frame information that has undergone the second screening, the key target in the image to be processed is determined.

[0055] The above process of determining the key target in the image to be processed may also include: firstly, based on multiple saliency information, sorting multiple detection frame information in descending order of saliency scores to obtain a second sorting result. Filtering the first predetermined number of detection frame information from the second sorting result. Then, based on multiple confidence information, filtering the detection frame information with a confidence score higher than a predetermined threshold from the first predetermined number of detection frame information to obtain the detection frame information corresponding to the key target.

[0056] Figure 3B The flowchart of an image processing method according to another embodiment of the present disclosure is schematically shown.

[0057] like Figure 3B As shown, the method may include the following operations S301 to S307, wherein operations S301 to S304 have been described in detail above and will not be repeated here. After determining the key target in the image to be processed, operation S305 is performed.

[0058] In operation S305 , based on the detection frame information of the key target, a sub-image including the key target is segmented from the image to be processed.

[0059] Then, in operation S306, search results related to the sub-image are obtained.

[0060] Next, in operation S307, the search result is output.

[0061] Reference below Figure 2 , Figure 4A and Figure 4B , the above-mentioned embodiments are exemplarily described.

[0062] Figure 4A An example schematic diagram of an optimization model according to an embodiment of the present disclosure is schematically shown.

[0063] Figure 4B An example schematic diagram of an image search process according to an embodiment of the present disclosure is schematically shown.

[0064] In this example, the client receives user input such as Figure 2 The image to be processed is uploaded to the server for image processing. The server obtains the image as shown in Figure 4A The optimized model 400 shown has completed training. Figure 4A As shown, the optimization model 400 includes: a detection framework 410 , a first output layer 420 , a second output layer 430 and a third output layer 440 .

[0065] The server inputs the image to be processed 450 to the input layer of the detection framework 410 of the optimization model 400, so that the first output layer 420, the second output layer 430 and the third output layer 440 of the optimization model 400 respectively output: Figure 2 The detection box information D{d1, d2, d3, d4} of detection boxes 1, 2, 3, and 4, Figure 2 The confidence information C{c1, c2, c3, c4} of the detection boxes 1, 2, 3, and 4, Figure 2 The saliency information S{s1, s2, s3, s4} of the detection boxes 1, 2, 3, and 4 in D is shown in FIG. 1 , where each element in D can be the coordinate information of the detection box, each element in C can be the confidence score, and each element in S can be the saliency score.

[0066] First, filter according to the confidence information C of the detection box, and set the filter confidence to conf_thres. If c i ≥conf_thres, the i-th detection frame is retained, otherwise the i-th detection frame is deleted. In this example, i is an integer greater than or equal to 1 and less than or equal to 4. In other examples, when n detection frames are detected, i is an integer greater than or equal to 1 and less than or equal to n, and n is a positive integer greater than 1. In this example, the detection frames retained are detection frames 3 and 4. Then, the retained detection frames are sorted in descending order according to the significance information S. The object target ranked first is the object target with the highest significance, and the object target is taken as the key target in the image to be processed. In this example, the significance score of detection frame 4 is higher than the significance score of detection frame 3, so the object target corresponding to detection frame 4 is determined to be the key target.

[0067] After determining the key target, the server can search based on the sub-image containing the key target to obtain multiple search results about the key target. The determined key target and the search results about the key target are sent to the client, which displays them to the user. The display interface is as follows Figure 4B As shown in the figure, the detection box of the key target (such as a red leather bag) is marked on the image to be processed, but the detection boxes of other objects are not marked. Multiple search results about the key target are displayed below, and the search results include relevant information such as pictures, URLs, and sources. Users can choose from multiple search results as needed.

[0068] According to an embodiment of the present disclosure, a model training method is also provided for training the optimization model of each embodiment above, and the method is described below through a legend. It should be noted that the sequence number of each operation in the following method is only used as a representation of the operation for the purpose of description, and should not be regarded as indicating the execution order of the various operations. Unless explicitly stated, the method does not need to be executed in the order shown.

[0069] Figure 5 The flowchart of the model training method according to the embodiment of the present disclosure is schematically shown. The model training method and the image processing method described above can be executed on the same side or on different sides, which is not limited here.

[0070] like Figure 5 As shown, the model training method may include the following operations S501 to S504.

[0071] In operation S501, a plurality of sample images are acquired.

[0072] In operation S502 , a plurality of regions are marked in each of a plurality of sample images, and a label is added to each of the plurality of regions.

[0073] Exemplarily, for a sample image, it is first necessary to perform region annotation in the sample image and mark multiple regions. Each region can be marked in the form of a labeling box. Then a label is added to each region, and the label includes a first label and a second label. The first label is used to characterize whether the region includes an object target, and the second label is used to characterize whether the object target contained in any region is significant. For example, for a region, a first label is added to the region. If the region contains an object target, the first label value is 1, and if the region does not contain an object target, the first label value is 0. And, a second label is added to the region. If the object target contained in the region is significant, the second label value is 1, and if the object target contained in the region is not significant, the second label value is 0.

[0074] Then, in operation S503, an initial model is obtained.

[0075] Among them, the initial model can be various target detection models based on deep learning, including R-CNN (Region Convolutional Neural Network), Fast R-CNN (Fast Region Convolutional Neural Network), Faster R-CNN (Faster Region Convolutional Neural Network), SSD (Single Shot MultiBox Detector), PVAnet (Performance Vs Accuracy), etc., without limitation. Since the target detection model usually includes two parallel output layers, an output layer is added to the initial model in this example to output saliency information. Exemplarily, the output layer of the initial model includes: a first output layer, a second output layer and a third output layer. The first output layer is used to output the predicted detection box information, the second output layer is used to output the predicted confidence information, and the third output layer is used to output the saliency information.

[0076] Next, in operation S504, the initial model is trained using the multiple sample images with labels obtained above to obtain an optimized model, and the optimized model is used to determine the key targets in the image to be processed.

[0077] According to an embodiment of the present disclosure, the process of training the initial model using labeled sample images to obtain an optimized model may include: inputting any sample image from a plurality of sample images into the input layer of the initial model, so that the output layer of the initial model outputs: multiple predicted detection box information about any sample image, multiple predicted confidence information for the multiple predicted detection box information, and multiple predicted significance information for the multiple predicted detection box information. Then, based on the difference between the multiple predicted detection box information and the multiple regions marked in any sample image, the difference between the multiple predicted confidence information and the first label of any sample image, and the difference between the multiple predicted significance information and the second label of any sample image, determine whether the loss function of the initial model has achieved convergence. If not, adjust the parameters of the initial model and repeat the above training process. If yes, use the initial model as the above optimized model. The predicted detection box may be referred to as the "prediction box" in the following text.

[0078] Exemplarily, the model training method according to the embodiment of the present disclosure adds the significant part into the loss function of the model training as the loss function of the model training process, and then performs end-to-end model training, and obtains the optimized model model A after multiple iterations. For example, the loss function of the model training is shown in formula (1).

[0079]

[0080] Where i is the label box number, and c i is the prediction confidence information of the i-th prediction box, c i * is the confidence information represented by the first label added to the i-th annotation box, d i is the coordinate information of the i-th prediction box, d i *The coordinate information of the i-th annotation box, s i is the saliency information of the i-th prediction box, s i * is the confidence information represented by the second label added to the i-th annotation box. Indicates c i and c i *The difference between Indicates d i and d i *The difference between Indicates i and i *The difference between.

[0081] N cls N represents the batch size of sample images used to train model parameters on confidence information. reg Represents the number of annotated regions (Anchor Location) in the sample image, N sal represents the batch size of sample images used to train model parameters about saliency information, and λ and μ represent penalty coefficients. For example, if the optimization model is more concerned with the accuracy of the detection box rather than the accuracy of saliency, then the value of μ should be smaller to reduce the excessive learning of saliency information by the entire network.

[0082] After multiple iterations of optimization, when the loss function shown in the above formula (1) reaches convergence, the training is determined to be finished. The initial model used at this time is used as the optimization model. The optimization model can be applied to the image processing method described in the above embodiments.

[0083] Figure 6 The block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure.

[0084] Figure 6As shown, the image processing device 600 may include: a first acquisition module 610 , a second acquisition module 620 , a detection module 630 , and a determination module 640 .

[0085] The first acquisition module 610 is used to acquire the image to be processed.

[0086] The second acquisition module 620 is used to acquire the optimization model.

[0087] The detection module 630 is used to input the image to be processed into the input layer of the optimization model so that the output layer of the optimization model outputs: multiple detection box information, multiple confidence information for the multiple detection box information, and multiple saliency information for the multiple detection box information.

[0088] The determination module 640 is used to determine the key target in the image to be processed based on multiple detection box information, multiple confidence information and multiple saliency information.

[0089] Figure 7 A block diagram of a model training device according to an embodiment of the present disclosure is schematically shown.

[0090] Figure 7 As shown, the model training device 700 may include: a third acquisition module 710 , a region annotation module 720 , a label adding module 730 , a fourth acquisition module 740 , and a training module 750 .

[0091] The third acquisition module 710 is used to acquire a plurality of sample images.

[0092] The region labeling module 720 is used to label multiple regions in each sample image of the multiple sample images.

[0093] The label adding module 730 is used to add a label to any area among the multiple areas.

[0094] The label includes a first label and a second label, the first label is used to indicate whether any region includes an object target, and the second label is used to indicate whether the object target contained in any region is significant.

[0095] The fourth acquisition module 740 is used to acquire an initial model.

[0096] The training module 750 is used to train the initial model using a plurality of sample images with labels to obtain an optimized model, and the optimized model is used to determine key targets in the image to be processed.

[0097] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module / unit / sub-unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0098] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0099] For example, any multiple of the first acquisition module 610, the second acquisition module 620, the detection module 630, and the determination module 640 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 610, the second acquisition module 620, the detection module 630, and the determination module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a suitable combination of any of them. Alternatively, at least one of the first acquisition module 610 , the second acquisition module 620 , the detection module 630 , and the determination module 640 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0100] Figure 8The block diagram schematically shows a computer device suitable for implementing the method described above according to an embodiment of the present disclosure. Figure 8 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 8 As shown, the computer device 800 according to the embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0102] In RAM 803, various programs and data required for the operation of device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other through bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0103] According to an embodiment of the present disclosure, the device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.

[0104] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0105] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0106] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0107] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0109] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing method, comprising: Get the image to be processed; Obtaining an optimization model, wherein the optimization model is trained based on an adjustable penalty coefficient, a difference between a plurality of predicted detection box information and a plurality of regions marked in any sample image, a difference between a plurality of predicted confidence information and a first label of any sample image, and a difference between a plurality of predicted saliency information and a second label of any sample image, wherein the first label is used to characterize whether any region includes an object target, and the second label is used to characterize whether the object target contained in any region is saliency; Inputting the image to be processed into the input layer of the optimization model, so that the output layer of the optimization model outputs: a plurality of detection frame information, a plurality of confidence information for the plurality of detection frame information, and a plurality of saliency information for the plurality of detection frame information, wherein the plurality of detection frame information represents the position information of a plurality of regions containing object targets detected by the optimization model in the image to be processed, the plurality of confidence information represents the confidence level of each of the plurality of regions, and the plurality of saliency information represents the saliency level of each of the plurality of regions; and Based on the multiple detection box information, the multiple confidence information and the multiple saliency information, determine the key target in the image to be processed, wherein the multiple confidence information is used to filter the multiple detection box information, and the multiple saliency information is used to sort the multiple detection box information.

2. The method according to claim 1, wherein: Each confidence information of the plurality of confidence information includes a confidence score, and each significance information of the plurality of significance information includes a significance score; The determining the key target in the image to be processed based on the multiple detection frame information, the multiple confidence information and the multiple saliency information includes: Based on the multiple pieces of confidence information, filter out detection box information having a confidence score higher than a predetermined threshold from the multiple pieces of detection box information to obtain first filtered detection box information; Based on the plurality of saliency information, sorting the first screened detection box information in descending order of saliency scores to obtain a sorting result; Filtering a predetermined number of detection box information from the sorting results to obtain second filtered detection box information; and Based on the second filtered detection frame information, a key target in the image to be processed is determined.

3. The method according to claim 1, further comprising: After determining the key target in the image to be processed, segmenting a sub-image containing the key target from the image to be processed based on the detection frame information of the key target; Obtaining search results related to the sub-image; and The search results are output.

4. A model training method, comprising: Acquire multiple sample images; marking a plurality of regions in each of the plurality of sample images; Adding a label to any area among the multiple areas, wherein the label includes a first label and a second label, the first label is used to indicate whether any area includes an object target, and the second label is used to indicate whether the object target included in any area is significant; Obtaining an initial model; and The initial model is trained using multiple sample images with the labels to obtain an optimized model, wherein the optimized model is used to determine key targets in the image to be processed, and the optimized model is obtained by training based on an adjustable penalty coefficient, differences between multiple predicted detection box information and multiple regions marked in any sample image, differences between multiple predicted confidence information and a first label of any sample image, and differences between multiple predicted significance information and a second label of any sample image, wherein the multiple predicted detection box information represents position information of multiple regions containing object targets detected by the optimized model in the sample image, the multiple predicted confidence information represents the confidence level of each of the multiple regions, the multiple predicted significance information represents the significance level of each of the multiple regions, the multiple predicted confidence information is used to screen the multiple predicted detection box information, and the multiple predicted significance information is used to sort the multiple predicted detection box information.

5. The method according to claim 4, wherein: The step of training the initial model using the sample images with the labels to obtain the optimized model includes: Inputting any sample image of the multiple sample images into the input layer of the initial model, so that the output layer of the initial model outputs: the multiple predicted detection box information about the any sample image, the multiple predicted confidence information for the multiple predicted detection box information, and the multiple predicted significance information for the multiple predicted detection box information; Determining whether the loss function of the initial model achieves convergence based on differences between the multiple predicted detection box information and the multiple regions marked in the any sample image, differences between the multiple predicted confidence information and the first label of the any sample image, and differences between the multiple predicted saliency information and the second label of the any sample image; If not, adjusting the parameters of the initial model and repeating the above training process; and If yes, the initial model is used as the optimized model.

6. The method according to claim 4, wherein: The output layer of the initial model includes: a first output layer, a second output layer and a third output layer; The first output layer is used to output predicted detection box information; The second output layer is used to output prediction confidence information; and The third output layer is used to output saliency information.

7. An image processing device, comprising: A first acquisition module, used for acquiring an image to be processed; a second acquisition module, configured to acquire an optimization model, wherein the optimization model is obtained by training based on an adjustable penalty coefficient, a difference between a plurality of predicted detection box information and a plurality of regions annotated in any sample image, a difference between a plurality of predicted confidence information and a first label of any sample image, and a difference between a plurality of predicted saliency information and a second label of any sample image, wherein the first label is used to characterize whether any region includes an object target, and the second label is used to characterize whether the object target contained in any region is saliency; a detection module, configured to input the image to be processed into an input layer of the optimization model, so that the output layer of the optimization model outputs: a plurality of detection frame information, a plurality of confidence information for the plurality of detection frame information, and a plurality of saliency information for the plurality of detection frame information, wherein the plurality of detection frame information represents the position information of a plurality of regions containing object targets detected by the optimization model in the image to be processed, the plurality of confidence information represents the confidence level of each of the plurality of regions, and the plurality of saliency information represents the saliency level of each of the plurality of regions; and A determination module is used to determine the key target in the image to be processed based on the multiple detection box information, the multiple confidence information and the multiple saliency information, wherein the multiple confidence information is used to filter the multiple detection box information, and the multiple saliency information is used to sort the multiple detection box information.

8. A model training device, comprising: A third acquisition module is used to acquire multiple sample images; A region marking module, used for marking a plurality of regions in each of the plurality of sample images; A label adding module, used to add a label to any area among the multiple areas, wherein the label includes a first label and a second label, the first label is used to indicate whether any area includes an object target, and the second label is used to indicate whether the object target included in any area is significant; A fourth acquisition module, used to acquire an initial model; and A training module is used to train the initial model using multiple sample images with the label to obtain an optimized model, wherein the optimized model is used to determine key targets in the image to be processed, and the optimized model is obtained by training based on an adjustable penalty coefficient, a difference between multiple predicted detection box information and multiple regions marked in any sample image, a difference between multiple predicted confidence information and a first label of any sample image, and a difference between multiple predicted significance information and a second label of any sample image, wherein the multiple predicted detection box information represents the location information of multiple regions containing object targets detected by the optimized model in the sample image, the multiple predicted confidence information represents the confidence level of each of the multiple regions, the multiple predicted significance information represents the significance level of each of the multiple regions, the multiple predicted confidence information is used to filter the multiple predicted detection box information, and the multiple predicted significance information is used to sort the multiple predicted detection box information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program: The method according to any one of claims 1 to 6.

10. A computer-readable storage medium having executable instructions stored thereon, which instructions, when executed by a processor, cause the processor to perform: The method according to any one of claims 1 to 6.

Images