Image detection model training method, image detection method, and related device

Abstract

The present application provides an image detection model training method, an image detection method, and a related device, which are used for improving the detection accuracy of image detection models and reducing manual labor costs. The method comprises: acquiring a training sample set, the training sample set comprising sample images of a plurality of categories; generating a plurality of texts on the basis of the plurality of categories, and extracting a text feature of each text; for each image in the training sample set, performing the following operations to determine a loss parameter of each image: inputting each image into a network model to be trained, and extracting from each image a first image feature of a region comprising a target object; determining a similarity between the first image feature and the text feature of each of the plurality of texts; and on the basis of a plurality of determined similarities corresponding to the first image feature, determining the loss parameter of each image; and on the basis of the loss parameters of all the images in the training sample set and a preset loss function, adjusting parameters of the network model.

Description

Image detection model training method, image detection method and related device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on August 3, 2023, with application number 202310974673.6 and invention name “A training method for an image detection model, an image detection method and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of target detection and discloses a training method for an image detection model, an image detection method and related devices. Background Art

[0004] In related technologies, a network is trained using pre-training samples to obtain a pre-trained network model. Real training samples are then input into the pre-trained network model for fine-tuning to obtain the final model. Pre-training samples are typically large in number and contain rich image features, such as color, shape, and texture. However, real training samples are typically smaller in number and have less robust image features. Fine-tuning the pre-trained network model using real training samples can be ineffective, limiting improvements in detection accuracy.

[0005] Summary of the Invention

[0006] The present application provides an image detection model training method, an image detection method and related devices, which are used to improve the detection accuracy of the image detection model and reduce labor costs.

[0007] In a first aspect, an embodiment of the present application provides a method for training an image detection model, comprising:

[0008] Obtaining a training sample set, the training sample set including sample images of multiple categories, one sample image corresponding to one category, and the label of the sample image being the category corresponding to the sample image;

[0009] Generating a plurality of texts according to the plurality of categories, and extracting text features of each text, wherein the plurality of texts correspond one-to-one to the plurality of categories, and each text includes the category corresponding to each text;

[0010] Performing the following operations for each image in the training sample set to determine a loss parameter for each image: inputting each image into a network model to be trained, extracting a first image feature of a region of each image containing a target object; determining a similarity between the first image feature and a text feature of each of the plurality of texts; and determining a loss parameter for each image based on a plurality of similarities corresponding to the determined first image feature;

[0011] Based on the loss parameters of all images in the training sample set and a preset loss function, the parameters of the network model to be trained are adjusted.

[0012] In one possible implementation, in the training method of the image detection model provided in the embodiment of the present application, generating multiple texts according to the multiple categories includes:

[0013] Generate each text according to at least one extended field and a category corresponding to each text, wherein the extended field includes one or more of the following:

[0014] The method for obtaining the images in the training sample set and the scenes to which the images in the training sample set belong.

[0015] In one possible implementation, in the training method of the image detection model provided in the embodiment of the present application, the training sample set includes: multiple sample subsets; the multiple sample subsets include a first sample subset and at least one second sample subset;

[0016] A method for acquiring sample images in the first sample subset is different from a method for acquiring sample images in the second sample subset;

[0017] Furthermore, the total number of sample images in the first sample subset is greater than the total number of sample images in each of the at least one second sample subset.

[0018] In one possible implementation, in the training method of the image detection model provided in the embodiment of the present application, in the loss function, the weight of the loss sub-function corresponding to the first sample subset is less than the weight corresponding to the loss sub-function corresponding to the at least one second sample subset.

[0019] For example, the loss function may be L=α M *Lpre+α Q *Lreal, wherein the Lpre represents the loss subfunction corresponding to the first sample subset, the α M represents the weight of the loss sub-function corresponding to the first sample subset; the Lreal represents the loss sub-function corresponding to the at least one second sample subset, the α QCharacterize the weight of the loss sub-function corresponding to the at least one second sample subset.

[0020] In some examples, the described Among them, the Characterizes the image features of the ath sample image in the first sample subset pre; the real_N characterizes the Nth second sample subset in the at least one second sample subset, the NT characterizes the total number of the at least one second sample subset, the Characterize the image features of the bth sample image in the Nth second sample subset.

[0021] described Characterizing a target text feature corresponding to an image feature of an a-th sample image in the first sample subset pre, wherein among the similarities between the image feature of the a-th sample image in the first sample subset pre and the text features of the plurality of texts, a similarity between the target text feature corresponding to the image feature of the a-th sample image in the first sample subset pre and the image feature of the a-th sample image in the first sample subset pre is the greatest;

[0022] described Characterizing the target text features corresponding to the image features of the bth sample image in the Nth second sample subset, among the similarities between the image features of the bth sample image in the Nth second sample subset and the text features of the multiple texts, the similarity between the target text features corresponding to the bth sample image in the Nth second sample subset and the image features of the bth sample image in the Nth second sample subset is the largest.

[0023] Characterization The loss parameter, Characterization The loss parameter is: The F te_k a k-th text feature among the text features representing the plurality of texts; Characterization With the The similarity of Characterization With the Similarity, τ is a constant.

[0024] In one possible embodiment, in the training method of the image detection model provided in the embodiment of the present application, the multiple sample subsets include multiple second sample subsets, wherein the scene to which the image in any second sample subset belongs is different from the scene to which the image in the multiple second sample subsets except any second sample subset belongs.

[0025] In one possible implementation, in the training method of the image detection model provided in an embodiment of the present application, the loss function includes a loss sub-function corresponding to the first sample subset and a loss sub-function corresponding to each second sample subset in the second sample subset;

[0026] The weight of the loss sub-function corresponding to the first sample subset is less than a minimum value of the weights of the loss sub-functions corresponding to the multiple second sample subsets.

[0027] For example, in the training method of the image detection model provided in the embodiment of the present application, the loss function is:

[0028] Wherein, the Lpre represents the loss sub-function corresponding to the first sample subset, the α M represents the weight of the loss sub-function corresponding to the first sample subset; the NT represents the total number of the plurality of second sample subsets, the Characterize the loss sub-function corresponding to the Nth second sample subset in the plurality of second sample subsets, the α real_N Characterize the loss sub-function corresponding to the Nth second sample subset.

[0029] In some examples, the Su Sohu Among them, the Characterizing the image features of the ath sample image in the first sample subset pre; Characterizing an image feature of the b-th sample image in the N-th second sample subset;

[0030] described Characterizing a target text feature corresponding to an image feature of an a-th sample image in the first sample subset pre, wherein among the similarities between the image feature of the a-th sample image in the first sample subset pre and the text features of the plurality of texts, a similarity between the target text feature corresponding to the image feature of the a-th sample image in the first sample subset pre and the image feature of the a-th sample image in the first sample subset pre is the greatest;

[0031] described Characterizing a target text feature corresponding to an image feature of a b-th sample image in the N-th second sample subset, wherein among the similarities between the image feature of the b-th sample image in the N-th second sample subset and the text features of the plurality of texts, the similarity between the target text feature corresponding to the b-th sample image in the N-th second sample subset and the image feature of the b-th sample image in the N-th second sample subset is the greatest;

[0032] Characterization The loss parameter, Characterization The loss parameter is: The F te_k a k-th text feature among the text features representing the plurality of texts; Characterization With the The similarity of Characterization With the Similarity, τ is a constant.

[0033] In a second aspect, an embodiment of the present application provides an image detection method, which may include:

[0034] Acquire the image to be tested;

[0035] If the image to be measured includes a target object, extracting image features of the area including the target object;

[0036] determining a similarity between the image feature and a text feature of each of a plurality of texts, wherein the plurality of texts are pre-generated and correspond one-to-one with a plurality of object categories, and each of the texts includes a corresponding object category;

[0037] The object category included in the target text is determined as the object category of the target object, wherein the similarity between the image feature and the text feature of the target text is the largest among the multiple similarities corresponding to the determined image feature.

[0038] In a possible implementation, an embodiment of the present application provides an image detection method, wherein the number of characters in each text is greater than a preset character number threshold.

[0039] In a possible implementation, an embodiment of the present application provides an image detection method, wherein the image to be detected is captured by an image capture device in a kitchen scene;

[0040] The plurality of object categories include:

[0041] Wearing chef hat and mask, wearing chef hat and not wearing mask, not wearing chef hat and wearing mask, not wearing chef hat and not wearing mask.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, which may include a memory and a processor;

[0043] The memory is used to store program instructions;

[0044] The processor is configured to execute the program instructions to implement the method as described in the first aspect and any possible implementation manner thereof, or to execute the method as described in the second aspect and any possible implementation manner thereof.

[0045] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which may include computer program instructions. When the computer program instructions are executed by a computer, the computer executes the method as described in the first aspect and any possible implementation manner thereof, or executes the method as described in the second aspect and any possible implementation manner thereof.

[0046] In a fifth aspect, an embodiment of the present application further provides a training device, comprising:

[0047] A sample set acquisition module is used to acquire a training sample set, wherein the training sample set includes sample images of multiple categories, one sample image corresponds to one category, and the label of the sample image is the category corresponding to the sample image;

[0048] a text feature generation module, configured to generate a plurality of texts according to the plurality of categories, and extract text features of each text, wherein the plurality of texts correspond one-to-one to the plurality of categories, and each text includes the category corresponding to the text;

[0049] A model training module is used to perform the following operations on each image in the training sample set to determine the loss parameters of each image: input each image into the network model to be trained, and extract the first image feature of the area of ​​each image containing the target object; determine the similarity between the first image feature and the text feature of each text in the multiple texts; and determine the loss parameter of each image based on the multiple similarities corresponding to the determined first image feature; and adjust the parameters of the network model to be trained based on the loss parameters of all images in the training sample set and a preset loss function.

[0050] In a sixth aspect, an embodiment of the present application further provides an image detection device, comprising:

[0051] An image acquisition module, used for acquiring an image to be tested;

[0052] An image detection module is used to extract image features of an area including a target object if the image to be tested includes the target object; determine the similarity between the image features and text features of each text in a plurality of texts, wherein the plurality of texts are pre-generated and correspond one-to-one with a plurality of object categories, and each text includes a corresponding object category; and determine the object category included in the target text as the object category of the target object, wherein the similarity between the image features and the text features of the target text is the largest among the multiple similarities corresponding to the determined image features.

[0053] The beneficial effects of the embodiments of the present application are as follows:

[0054] The present application provides a training method for an image detection model, an image detection method, and related devices, which can generate extended texts of various categories by using the categories of sample images in a training sample set. The network model is trained using the text features of the extended text and the image features of the sample images. This method can increase the amount of data in the training process, improve the robustness of the model, enrich the features of the sample images, have good detection accuracy, and do not require a large number of real training samples, thereby reducing the cost of manual annotation. There is no situation where the feature distribution of the pre-training samples and the real training samples is different, resulting in poor training network model results.

[0055] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 shows the training process of related technologies;

[0058] FIG2 is a schematic flow chart of a method for training an image detection model according to an embodiment of the present application;

[0059] FIG3 is a schematic diagram of a training process according to an embodiment of the present application;

[0060] FIG4 is a schematic diagram of a pre-training sample image;

[0061] FIG5 is a schematic diagram of a sample image of a real scene;

[0062] FIG6 is a schematic diagram of a training process provided in an embodiment of the present application;

[0063] FIG7 is a schematic flow chart of an image detection method provided in an embodiment of the present application;

[0064] FIG8 is a schematic structural diagram of a training device provided in an embodiment of the present application;

[0065] FIG9 is a schematic structural diagram of a detection device provided in an embodiment of the present application;

[0066] FIG10 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0067] FIG11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Unless there is a conflict, the embodiments in the present application and the features in the embodiments can be combined with each other in any way. In addition, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that here.

[0069] The terms "first" and "second" in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of its variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in this application can mean at least two, for example, two, three or more, and the embodiments of this application are not limited thereto.

[0070] In the technical solution of this application, the collection, dissemination, and use of image data comply with relevant national laws and regulations.

[0071] Before introducing a training method for an image detection model provided in an embodiment of the present application, for ease of understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0072] In related art, as shown in Figure 1, a network model is trained using a set of pretrained samples to obtain a pretrained network model. Each pretrained sample is input into the network, and the network is trained with the labels of the output samples as the goal to obtain the pretrained network model. The pretrained network model is then fine-tuned using real training samples. Each real training sample is input into the pretrained network model, and the pretrained network model is adjusted with the labels of the output samples as the goal to obtain the final detection model.

[0073] Typically, pre-training sample sets consist of pre-acquired, annotated images, and they contain relatively rich image features. However, real-world training sample sets consist of images used in actual model applications, and image annotation requires high labor costs, resulting in a smaller number of samples in the real-world training sample set. Furthermore, the image features in the real-world training sample set are often poor. Consequently, fine-tuning the pre-trained network model on real-world sample images yields suboptimal results, limiting improvements in detection accuracy.

[0074] In view of this, embodiments of the present application provide a training method for an image detection model, an image detection method, and related devices. Labels of sample images are expanded into text, and text features are extracted. A network model is trained by combining text features with image features. This allows the image detection model to be trained by integrating semantic and image information, thereby improving detection accuracy.

[0075] FIG2 shows a method for training an image detection model according to an exemplary embodiment. The method may be performed by an electronic device and may include the following steps:

[0076] S201 : Acquire a training sample set, where the training sample set includes sample images of multiple categories, one sample image corresponds to one category, and a label of the sample image is the category corresponding to the sample image.

[0077] In specific implementation, as shown in Figure 3, when the electronic device performs image detection model training, the training sample set can be input into the network model, and the results can be output. The network model parameters can be adjusted according to the output results to implement the process of training the network model. In the training method of the image detection model provided in the embodiment of the present application, no pre-trained network model is generated.

[0078] The training sample set acquired by the electronic device may include sample images from a variety of sources, including the aforementioned pre-training sample images and real-world sample images. For the sake of convenience in introducing the method provided in this application, we will use the kitchen as an example, with the detection objective of detecting whether a chef in the kitchen is wearing a chef's hat and mask. It should be noted that the training method provided in this application can also be applied to other scenarios and for other detection purposes.

[0079] As shown in Figure 4, pre-training sample images are usually processed images with rich image features. However, real scene sample images are usually collected in application scenarios, as shown in Figure 5, and have poor image features.

[0080] The training sample set may include sample images of multiple categories, wherein one sample image corresponds to one category, and the category corresponding to the sample image is used as the label of the sample image. For example, multiple categories may be recorded as: wearing a chef's hat and wearing a mask, wearing a chef's hat and not wearing a mask, not wearing a chef's hat and wearing a mask, not wearing a chef's hat and not wearing a mask. For another example, multiple categories may be recorded as: chef's hat+mask, chef's hat+nomask, no chef's hat+mask, no chef's hat+nomask. The embodiment of the present application does not specifically limit the language and form of the image label, but can respectively represent the meaning corresponding to each category recorded in the Chinese language.

[0081] S202 : generating a plurality of texts according to the plurality of categories, and extracting text features of each text, wherein the plurality of texts correspond one-to-one to the plurality of categories, and each text includes the category corresponding to the text.

[0082] During specific implementation, the electronic device can expand each category and generate corresponding text. Optionally, the electronic device can generate each text based on at least one extended field and the category corresponding to each text, wherein the extended field includes one or more of the following: the acquisition method of the image in the training sample set, and the scene to which the image in the training sample set belongs. The image acquisition method can represent the source of the image. The scene to which the image belongs can represent the application scenario of the network model.

[0083] In some examples, the text generated by the electronic device after expanding each category may include the method of obtaining the sample image. For example, the text generated after expanding each category is:

[0084] Text 1: An online image of a person wearing a chef's hat and wearing a mask;

[0085] Text 2: An online image of a person not wearing a chef's hat and wearing a mask;

[0086] Text 3: A real photo of a person wearing a chef's hat and not wearing a mask;

[0087] Text 4: A real photo of a person not wearing a chef's hat and wearing a mask.

[0088] Among them, “online” represents a way of acquiring images, and “real” represents a way of acquiring images.

[0089] In other examples, the text generated by the electronic device after expanding each category may include the method of obtaining the sample image and the scene to which the image belongs. For example, the text generated after expanding each category is:

[0090] Text 1: An online image of a person wearing a chef's hat and wearing a mask in sceneX;

[0091] Text 2: An online image of a person not wearing a chef's hat and wearing a mask in sceneX;

[0092] Text 3: A real photo of a person wearing a chef's hat and not wearing a mask in sceneX;

[0093] Text 4: A real photo of a person not wearing a chef's hat and wearing a mask in sceneX.

[0094] Among them, "online" represents a method of acquiring images, and "real" represents a method of acquiring images. "sceneX" represents the scene X to which the image belongs, that is, the image is collected (or shot) in scene X. For example, "scene1" can represent the scene 1 to which the image belongs, and "scene2" can represent the scene 2 to which the image belongs. The methods of acquiring images in this application may include but are not limited to: online acquisition, generation, acquisition, etc. In the above example, "online" can represent online acquisition. "real" can represent acquisition.

[0095] The electronic device can extract text features of the generated text, and can obtain text feature 1 of text 1, text feature 2 of text 2, text feature 3 of text 3, and text feature 4 of text 4.

[0096] Optionally, the number of characters in the text generated by the electronic device is greater than or equal to a preset character threshold. For example, the preset character threshold may be 16 characters. Optionally, the number of characters in the text generated by the electronic device may be greater than or equal to 128 characters.

[0097] S203, performing the following operations for each image in the training sample set to determine the loss parameter of each image: inputting each image into the network model to be trained, extracting the first image feature of the area of ​​each image containing the target object; determining the similarity between the first image feature and the text feature of each text in the multiple texts; and determining the loss parameter of each image based on the multiple similarities corresponding to the determined first image feature.

[0098] S204: Adjust the parameters of the network model to be trained based on the loss parameters of all images in the training sample set and a preset loss function.

[0099] In a possible implementation, during step 203 and step 204, the loss function configured in the electronic device may not distinguish between the acquisition methods of the sample images. The electronic device may be configured with a first loss function: L = ∑ i∈训练样本集合 -logp(F im_i ,F tem_i );

[0100] Among them, the F im_i Characterize the image features of the i-th sample image in the training sample set, the F te_k The kth text feature among the text features representing the plurality of texts; the F tem_i Characterizing a target text feature corresponding to the i-th sample image, wherein among the similarities between the image feature of the i-th sample image and the text features of the plurality of texts, the similarity between the target text feature corresponding to the i-th sample image and the image feature of the i-th sample image is the greatest;

[0101] The p(F im_i ,F tem_i ) characterizes the F im_i The loss parameter, S(F im_i ,F tem_i ) characterizes the F im_i With the F tem_i The similarity of S(F im_i ,Fte_k ) characterizes the F im_i With the F te_k Similarity, τ is a constant.

[0102] For each sample image, the electronic device may determine the similarity between the sample image and each text feature, for example, Among them, F im Characterize any image feature, F te Represents any text feature.

[0103] For each sample image, the electronic device may determine the text feature corresponding to the maximum value of the similarities between the image feature of the sample image and each text feature as the target text feature corresponding to the image feature of the sample image. And based on the similarities between the image feature of the sample image and each text feature, and the target text feature corresponding to the image feature of the sample image, determine the loss parameter of the sample image. For example, the F im_i The loss parameter is:

[0104] Based on the first loss function and the loss parameter for each sample image, the electronic device can determine a loss value for this training, adjust the model based on the loss value, and then repeat the training process. Optionally, the electronic device can terminate the training after determining that the loss value is less than a preset loss value threshold, and use the trained model as the trained image detection model.

[0105] In another possible embodiment, during steps 203 and 204, the loss function configured in the electronic device can distinguish the acquisition method of each sample image and the scene to which the image belongs. For example, the training sample set may include: multiple sample subsets. The multiple sample subsets may include a first sample subset and at least one second sample subset. The acquisition method of the sample images in the first sample subset is different from the acquisition method of the sample images in the second sample subset. As an example, the acquisition method of the sample images in the first sample subset may be acquired or generated online, and the acquisition method of the sample images in the second sample subset may be collected. The total number of sample images in the first sample subset is greater than the total number of sample images in each of the at least one second sample subset.

[0106] In some examples, the multiple sample subsets include at least one second sample subset. When the at least one second sample subset is a plurality of second sample subsets, the scene to which images in any second sample subset belong is different from the scene to which images in the at least one second sample subset other than the second sample subset belong. For example, the second sample subsets may include second sample subset 1 and second sample subset 2. The scene to which images in second sample subset 1 belong may be scene 1. The scene to which images in second sample subset 2 belong may be scene 2.

[0107] The electronic device may be configured with a second loss function:

[0108] L=α M *Lpre+α Q *Lreal

[0109] Wherein, the Lpre represents the loss sub-function corresponding to the first sample subset, the α M represents the weight of the loss sub-function corresponding to the first sample subset; the Lreal represents the loss sub-function corresponding to the at least one second sample subset, the α Q Characterize the weight of the loss sub-function corresponding to the at least one second sample subset.

[0110] Optionally, the described Among them, the Characterizes the image features of the ath sample image in the first sample subset pre; the real_N characterizes the Nth second sample subset in the at least one second sample subset, the NT characterizes the total number of the at least one second sample subset, the Characterize the image features of the bth sample image in the Nth second sample subset.

[0111] described Characterizing a target text feature corresponding to an image feature of an a-th sample image in the first sample subset pre, wherein among the similarities between the image feature of the a-th sample image in the first sample subset pre and the text features of the plurality of texts, a similarity between the target text feature corresponding to the image feature of the a-th sample image in the first sample subset pre and the image feature of the a-th sample image in the first sample subset pre is the greatest;

[0112] described Characterizing the target text features corresponding to the image features of the bth sample image in the Nth second sample subset, among the similarities between the image features of the bth sample image in the Nth second sample subset and the text features of the multiple texts, the similarity between the target text features corresponding to the bth sample image in the Nth second sample subset and the image features of the bth sample image in the Nth second sample subset is the largest.

[0113] Characterization The loss parameter, Characterization The loss parameter is: The F te_k a k-th text feature among the text features representing the plurality of texts; Characterization With the The similarity of Characterization With the Similarity, τ is a constant.

[0114] For each sample image in each sample subset, the electronic device may determine the text feature corresponding to the maximum value of the similarities between the image feature of the sample image and each text feature as the target text feature corresponding to the image feature of the sample image. Furthermore, based on the similarities between the image feature of the sample image and each text feature, and the target text feature corresponding to the image feature of the sample image, the electronic device may determine a loss parameter for the sample image.

[0115] For example The loss parameter For example Loss parameters

[0116] Based on the second loss function and the loss parameter for each sample image in each sample subset, the electronic device can determine a loss value for this training, adjust the model based on the loss value, and then repeat the training process. Optionally, the electronic device can terminate the training after determining that the loss value is less than a preset loss value threshold, and use the trained model as the trained image detection model.

[0117] In a possible design, in the above second loss function, the weight α corresponding to the loss sub-function Lpre corresponding to the first sample subset is M Less than the weight α corresponding to the loss sub-function Lreal corresponding to the at least one second sample subset QIn such a design, the detection effect of the image detection model on the image obtained by the acquisition method corresponding to the second sample subset can be improved.

[0118] In some other examples, the multiple sample subsets include multiple second sample subsets, wherein the scene to which images in any second sample subset belong is different from the scene to which images in the multiple second sample subsets other than the second sample subset belong. For example, the second sample subsets may include second sample subset 1 and second sample subset 2. The scene to which images in second sample subset 1 belong may be scene 1. The scene to which images in second sample subset 2 belong may be scene 2.

[0119] In electronic devices, a third loss function can be configured:

[0120] Wherein, the Lpre represents the loss sub-function corresponding to the first sample subset, the α M represents the weight of the loss sub-function corresponding to the first sample subset; the NT represents the total number of the plurality of second sample subsets, the Characterize the loss sub-function corresponding to the Nth second sample subset in the plurality of second sample subsets, the α real_N Characterize the loss sub-function corresponding to the Nth second sample subset.

[0121] Optionally, the described Among them, the Characterizing the image features of the ath sample image in the first sample subset pre; Characterizing an image feature of the b-th sample image in the N-th second sample subset;

[0122] described Characterizing a target text feature corresponding to an image feature of an a-th sample image in the first sample subset pre, wherein among the similarities between the image feature of the a-th sample image in the first sample subset pre and the text features of the plurality of texts, a similarity between the target text feature corresponding to the image feature of the a-th sample image in the first sample subset pre and the image feature of the a-th sample image in the first sample subset pre is the greatest;

[0123] described Characterizing a target text feature corresponding to an image feature of a b-th sample image in the N-th second sample subset, wherein among the similarities between the image feature of the b-th sample image in the N-th second sample subset and the text features of the plurality of texts, the similarity between the target text feature corresponding to the b-th sample image in the N-th second sample subset and the image feature of the b-th sample image in the N-th second sample subset is the greatest;

[0124] Characterization The loss parameter, Characterization The loss parameter is: The F te_k a k-th text feature among the text features representing the plurality of texts; Characterization With the The similarity of Characterization With the Similarity, τ is a constant.

[0125] For each sample image in each sample subset, the electronic device may determine the text feature corresponding to the maximum value of the similarities between the image feature of the sample image and each text feature as the target text feature corresponding to the image feature of the sample image. Furthermore, based on the similarities between the image feature of the sample image and each text feature, and the target text feature corresponding to the image feature of the sample image, the electronic device may determine a loss parameter for the sample image.

[0126] For example The loss parameter For example Loss parameters

[0127] Based on the third loss function and the loss parameter for each sample image in each sample subset, the electronic device can determine a loss value for this training, adjust the model based on the loss value, and then repeat the training process. Optionally, the electronic device can terminate the training after determining that the loss value is less than a preset loss value threshold, and use the trained model as the trained image detection model.

[0128] In a possible design, in the third loss function, the weight α of the loss sub-function Lpre corresponding to the first sample subset is M is smaller than the loss sub-function corresponding to each second sample subset in the plurality of second sample subsets The weight α real_N (where N takes the minimum value of 1-NT). For example, the plurality of second sample subsets may include second sample subset 1 and second sample subset 2. The loss subfunction corresponding to the second sample subset 1 is The weight is denoted as α real_1 , the loss sub-function corresponding to the second sample subset 2 The weight is denoted as α real_2 α real_1and α real_2 The minimum value in is greater than the weight α of the loss sub-function Lpre corresponding to the first sample subset M of.

[0129] Optionally, in the third loss function, the weight of the loss sub-function corresponding to each second sample subset can be configured based on the actual application scenario. In some examples, the weight of the loss sub-function corresponding to each second sample subset can be determined based on the number of samples in each second sample subset. For example, the weight of the loss sub-function corresponding to a second sample subset with a large number of samples can be smaller than the weight of the loss sub-function corresponding to a second sample subset with a small number of samples.

[0130] FIG6 exemplarily shows a training process of an image detection model. The electronic device may obtain a training sample set, wherein the training sample set may include a first sample subset and a second sample subset. Optionally, the first sample subset may be a pre-training sample set I in the related art. pre The second sample subset can be the real training set I in the related art real .

[0131] The electronic device can use a Region Proposal Network (RPN) to find the target object. For example, in the aforementioned kitchen scene, the target object may be a person. The region of interest found by the RPN contains the target object. The electronic device can use an image encoder to generate image features of the region of interest of each sample image.

[0132] The electronic device may generate extended text for each label based on the labels of the samples in the training sample set. Optionally, the electronic device may further generate extended text for each label based on the method for acquiring the samples in the first sample subset, the method for acquiring the samples in the second sample subset, and the scenarios to which the samples in the second sample subset belong. The electronic device may generate text features for each text using a text encoder.

[0133] For the image features of each sample image, the electronic device can calculate the similarity between each image feature and each text feature, and determine the loss value based on a preset loss function to adjust the network parameters of the training model.

[0134] In some examples, the training sample set may include a first sample subset pre and a second sample subset real_1. The electronic device may be configured with a fourth loss function: L=0.3*Lpre+0.7*Lreal.

[0135] In the fourth loss function, described The weight of the loss sub-function corresponding to the first sample subset is 0.3, and the weight of the loss sub-function corresponding to the second sample subset real_1 is 0.7.

[0136] Among them, the Characterizes the image features of the ath sample image in the first sample subset pre, the real_N characterizes the Nth second sample subset in the plurality of second sample subsets, the Characterize the image features of the b-th sample image in the second sample subset real_1; Characterizing a target text feature corresponding to an image feature of an a-th sample image in the first sample subset pre, wherein among the similarities between the image feature of the a-th sample image in the first sample subset pre and the text features of the plurality of texts, a similarity between the target text feature corresponding to the image feature of the a-th sample image in the first sample subset pre and the image feature of the a-th sample image in the first sample subset pre is the greatest;

[0137] described Characterizing a target text feature corresponding to an image feature of the b-th sample image in the second sample subset real_1, wherein among the similarities between the image feature of the b-th sample image in the second sample subset real_1 and the text features of the plurality of texts, the similarity between the target text feature corresponding to the b-th sample image in the second sample subset real_1 and the image feature of the b-th sample image in the second sample subset real_1 is the greatest;

[0138] Characterization The loss parameter, Characterization The loss parameter is: The F te_k a kth text feature among the text features representing the plurality of texts; Characterization With the The similarity of Characterization With the Similarity, τ is a constant.

[0139] In some examples, the training sample set may include a first sample subset pre, a second sample subset real_1, and a second sample subset real_2. The electronic device may be configured with a fifth loss function:

[0140] L=0.2*Lpre+0.4*Lreal

[0141] In the fifth loss function, described NT = 2. The weight of the loss sub-function corresponding to the first sample subset is 0.2, and the weight of the loss sub-function corresponding to the second sample subset is 0.4.

[0142] Among them, the Characterizes the image features of the ath sample image in the first sample subset pre, the real_N characterizes the Nth second sample subset in the plurality of second sample subsets, the Characterize the image features of the b-th sample image in the second sample subset real_1; Characterizing a target text feature corresponding to an image feature of an a-th sample image in the first sample subset pre, wherein among the similarities between the image feature of the a-th sample image in the first sample subset pre and the text features of the plurality of texts, a similarity between the target text feature corresponding to the image feature of the a-th sample image in the first sample subset pre and the image feature of the a-th sample image in the first sample subset pre is the greatest;

[0143] described Characterizing a target text feature corresponding to an image feature of the b-th sample image in the second sample subset real_1, wherein among the similarities between the image feature of the b-th sample image in the second sample subset real_1 and the text features of the plurality of texts, the similarity between the target text feature corresponding to the b-th sample image in the second sample subset real_1 and the image feature of the b-th sample image in the second sample subset real_1 is the greatest;

[0144] Characterization The loss parameter, Characterization The loss parameter is: The F te_k a k-th text feature among the text features representing the plurality of texts; Characterization With the The similarity of Characterization With the Similarity, τ is a constant.

[0145] described Characterizing the image features of the c-th sample image in the N-th second sample subset; Characterizing a target text feature corresponding to an image feature of the c-th sample image in the second sample subset real_2, wherein among the similarities between the image feature of the c-th sample image in the second sample subset real_2 and the text features of the plurality of texts, the similarity between the target text feature corresponding to the c-th sample image in the second sample subset real_2 and the image feature of the c-th sample image in the second sample subset real_2 is the largest;

[0146] Characterization The loss parameter, Characterization With the similarity.

[0147] FIG7 exemplarily illustrates an image detection method that can be performed by a processor or an electronic device. The method may include:

[0148] S701: Acquire an image to be tested.

[0149] The image detection method provided in this application is described using a processor as an example. In some possible scenarios, the processor is configured with a pre-trained image detection model. Optionally, the image detection model training process can be performed in other electronic devices, and the processor stores the trained image detection model. Alternatively, the image detection model training process is performed in the processor, and the processor can use the trained image detection model to perform detection tasks.

[0150] S702: If the image to be measured includes a target object, extract image features of the area including the target object.

[0151] In a specific implementation, the processor may use the RPN technology to obtain a region of interest, that is, a region including the target object, and use the features of the region as image features of the image to be measured.

[0152] S703, determining a similarity between the image feature and a text feature of each of a plurality of texts, wherein the plurality of texts are pre-generated, and the plurality of texts correspond one-to-one to a plurality of object categories, and each of the texts includes a corresponding object category.

[0153] In a specific implementation, the multiple texts pre-generated by the electronic device are respectively extended texts of multiple object categories. Each text may include a corresponding object category. The electronic device may extract text features of each text.

[0154] S704 : Determine the object category included in the target text as the object category of the target object, wherein the similarity between the image feature and the text feature of the target text is the largest among the multiple similarities corresponding to the determined image feature.

[0155] In a specific implementation, the electronic device may determine the text corresponding to the text feature with the greatest similarity to the image feature of the image to be tested as the target text, and determine the object category contained in the target text as the object category of the image to be tested, thereby detecting the object category corresponding to the image to be tested.

[0156] In some examples, the number of characters in the text generated by the electronic device is greater than or equal to a preset character threshold. For example, the preset character threshold may be 16 characters. Optionally, the number of characters in the text generated by the electronic device may be greater than or equal to 128 characters.

[0157] In some examples, the image to be tested is captured by an image acquisition device in a kitchen scene; the target object may be a person. The multiple object categories include: wearing a chef's hat and a mask, wearing a chef's hat and not wearing a mask, not wearing a chef's hat and wearing a mask, and not wearing a chef's hat and not wearing a mask.

[0158] Based on the same technical concept, the embodiment of the present application provides a training device that can achieve the same technical effects as the aforementioned training method, which will not be described in detail here. Referring to Figure 8, the device includes a sample set acquisition module, a text feature generation module, and a model training module. Among them:

[0159] A sample set acquisition module is used to acquire a training sample set, wherein the training sample set includes sample images of multiple categories, one sample image corresponds to one category, and the label of the sample image is the category corresponding to the sample image;

[0160] a text feature generation module, configured to generate a plurality of texts according to the plurality of categories, and extract text features of each text, wherein the plurality of texts correspond one-to-one to the plurality of categories, and each text includes the category corresponding to the text;

[0161] A model training module is used to perform the following operations on each image in the training sample set to determine the loss parameters of each image: input each image into the network model to be trained, and extract the first image feature of the area of ​​each image containing the target object; determine the similarity between the first image feature and the text feature of each text in the multiple texts; and determine the loss parameter of each image based on the multiple similarities corresponding to the determined first image feature; and adjust the parameters of the network model to be trained based on the loss parameters of all images in the training sample set and a preset loss function.

[0162] In one possible implementation, the text feature generation module is specifically configured to generate each text based on at least one extended field and a category corresponding to each text, wherein the extended field includes one or more of the following:

[0163] The method for obtaining the images in the training sample set and the scenes to which the images in the training sample set belong.

[0164] In a possible implementation, the training sample set includes: a plurality of sample subsets; the plurality of sample subsets include a first sample subset and at least one second sample subset;

[0165] A method for acquiring sample images in the first sample subset is different from a method for acquiring sample images in the second sample subset;

[0166] Furthermore, the total number of sample images in the first sample subset is greater than the total number of sample images in each of the at least one second sample subset.

[0167] In one possible implementation, in the training method of the image detection model provided in the embodiment of the present application, in the loss function, the weight of the loss sub-function corresponding to the first sample subset is less than the weight corresponding to the loss sub-function corresponding to the at least one second sample subset.

[0168] In one possible embodiment, in the training method of the image detection model provided in the embodiment of the present application, the multiple sample subsets include multiple second sample subsets, wherein the scene to which the image in any second sample subset belongs is different from the scene to which the image in the multiple second sample subsets except any second sample subset belongs.

[0169] In one possible implementation, in the training method of the image detection model provided in an embodiment of the present application, the loss function includes a loss sub-function corresponding to the first sample subset and a loss sub-function corresponding to each second sample subset in the second sample subset;

[0170] The weight of the loss sub-function corresponding to the first sample subset is less than a minimum value of the weights of the loss sub-functions corresponding to the multiple second sample subsets.

[0171] Based on the same technical concept, the present application embodiment provides a training device that can achieve the same technical effects as the above detection method, which will not be described in detail here. Referring to Figure 9, the device includes an image acquisition module and an image detection module.

[0172] An image acquisition module, used for acquiring an image to be tested;

[0173] An image detection module is used to extract image features of an area including a target object if the image to be tested includes the target object; determine the similarity between the image features and text features of each text in a plurality of texts, wherein the plurality of texts are pre-generated and correspond one-to-one with a plurality of object categories, and each text includes a corresponding object category; and determine the object category included in the target text as the object category of the target object, wherein the similarity between the image features and the text features of the target text is the largest among the multiple similarities corresponding to the determined image features.

[0174] In a possible implementation, the number of characters in each text is greater than a preset character number threshold.

[0175] In a possible implementation, the image to be measured is captured by an image capture device in a kitchen scene;

[0176] The plurality of object categories include:

[0177] Wearing chef hat and mask, wearing chef hat and not wearing mask, not wearing chef hat and wearing mask, not wearing chef hat and not wearing mask.

[0178] Based on the same technical concept, an embodiment of the present application provides a first electronic device, which can execute the training method provided in the above embodiment and achieve the same technical effect, which will not be repeated here.

[0179] Referring to Figure 10, the first electronic device includes a processor 1001, a memory 1002 and a communication interface 1003. The processor 1001, the memory 1002 and the communication interface are connected via a bus 1004. The communication interface 1003 is used to communicate with other electronic devices, including but not limited to interactive sample images. The memory 1002 stores a computer program, and the processor 1001 executes the steps in the training method in the above embodiment according to the computer program.

[0180] The processor involved in Figure 10 of the embodiment of the present application can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0181] Based on the same technical concept, an embodiment of the present application provides a second electronic device, which can execute the training method provided in the above embodiment and achieve the same technical effect, which will not be repeated here.

[0182] Referring to Figure 11, the second electronic device includes a processor 1101, a memory 1102 and a communication interface 1103. The processor 1101, the memory 1102 and the communication interface are connected via a bus 1104. The communication interface 1103 is used to communicate with other electronic devices, including but not limited to interacting with images to be tested. The memory 1102 stores a computer program, and the processor 1101 executes the steps in the detection method in the above embodiment according to the computer program.

[0183] Optionally, the second electronic device is the same device as the first electronic device. Alternatively, the second electronic device is different from the first electronic device, wherein the second electronic device may store an image detection model trained by the first electronic device. The second electronic device may exchange relevant parameters of the image detection model with the first electronic device.

[0184] The processor involved in Figure 11 of the embodiment of the present application can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0185] In addition, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the training method or image detection method provided by any embodiment of the present application.

[0186] An embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a computer, implements the steps of the training method or image detection method provided in any of the above embodiments.

[0187] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0188] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.

[0189] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0191] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A training method for an image detection model, characterized in that: The method comprises: Acquire a training sample set, the training sample set comprising sample images of multiple categories, one sample image corresponds to one category, and the label of the sample image is the category corresponding to the sample image; According to the multiple categories, multiple texts are generated, and text features of each text are extracted, wherein the multiple texts correspond to the multiple categories one by one, and each text includes the category corresponding to each text; For each image in the training sample set, the following operations are performed to determine the loss parameter of each image: each image is input into the network model to be trained, and a first image feature of a region of each image containing a target object is extracted; a similarity between the first image feature and a text feature of each of the multiple texts is determined; and a loss parameter of each image is determined based on multiple similarities corresponding to the determined first image feature; Based on the loss parameters of all images in the training sample set and a preset loss function, the parameters of the network model to be trained are adjusted.

2. The method according to claim 1, characterized in that The step of generating a plurality of texts according to the plurality of categories comprises: Generate each text according to at least one extended field and a category corresponding to each text, wherein the extended field includes one or more of the following: The method for acquiring the images in the training sample set and the scenes to which the images in the training sample set belong.

3. The method according to claim 1 or 2, characterized in that The training sample set includes: a plurality of sample subsets; the plurality of sample subsets include a first sample subset and at least one second sample subset; The method for acquiring the sample images in the first sample subset is different from the method for acquiring the sample images in the second sample subset; Furthermore, the total number of sample images in the first sample subset is greater than the total number of sample images in each of the at least one second sample subset.

4. The method according to claim 3, characterized in that In the loss function, the weight of the loss sub-function corresponding to the first sample subset is less than the weight corresponding to the loss sub-function corresponding to the at least one second sample subset.

5. The method according to claim 3, characterized in that The multiple sample subsets include multiple second sample subsets, wherein the scene to which the images in any second sample subset belong is different from the scene to which the images in the multiple second sample subsets except the one in the second sample subset belong.

6. The method according to claim 5, characterized in that The loss function includes a loss sub-function corresponding to the first sample subset and a loss sub-function corresponding to each second sample subset in the second sample subset; The weight of the loss sub-function corresponding to the first sample subset is less than the minimum value of the weights of the loss sub-functions corresponding to the multiple second sample subsets.

7. An image detection method, characterized in that: The method comprises: Acquire the image to be tested; If the image to be measured includes a target object, extracting image features of a region including the target object; Determining a similarity between the image feature and a text feature of each of a plurality of texts, wherein the plurality of texts are pre-generated, and the plurality of texts correspond one-to-one to a plurality of object categories, and each of the texts includes a corresponding object category; The object category included in the target text is determined as the object category of the target object, wherein the similarity between the image feature and the text feature of the target text is the largest among the multiple similarities corresponding to the determined image feature.

8. The method according to claim 7, characterized in that The number of characters in each text is greater than a preset character number threshold.

9. The method according to claim 7 or 8, characterized in that The image to be tested is collected by an image acquisition device in a kitchen scene; The plurality of object categories include: Wearing chef hat and mask, wearing chef hat and not wearing mask, not wearing chef hat and wearing mask, not wearing chef hat and not wearing mask.

10. An electronic device, characterized in that: including memory and processor; The memory is used to store program instructions; The processor is used to execute the program instructions to implement the method according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computer, the computer performs the method according to any one of claims 1 to 9.

12. A training device, characterized in that: include: A sample set acquisition module is used to acquire a training sample set, wherein the training sample set includes sample images of multiple categories, one sample image corresponds to one category, and the label of the sample image is the category corresponding to the sample image; A text feature generation module, used to generate a plurality of texts according to the plurality of categories, and extract text features of each text, wherein the plurality of texts correspond to the plurality of categories one by one, and each text includes the category corresponding to the text; The model training module is used to perform the following operations on each image in the training sample set to determine the loss parameter of each image: input each image into the network model to be trained, and extract the first image feature of the area of ​​each image containing the target object; Determine the similarity between the first image feature and the text feature of each of the multiple texts; and determine the loss parameter of each image based on the multiple similarities corresponding to the determined first image feature; and adjust the parameters of the network model to be trained based on the loss parameters of all images in the training sample set and a preset loss function.

13. An image detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be tested; An image detection module is used to extract image features of an area including a target object if the image to be tested includes a target object; determine the similarity between the image features and text features of each of a plurality of texts, wherein the plurality of texts are pre-generated and correspond one-to-one to a plurality of object categories, and each of the texts includes a corresponding object category; determine the object category included in the target text as the object category of the target object, wherein the similarity between the image features and the text features of the target text is the largest among the plurality of similarities corresponding to the determined image features.

Images