Model training method, image category determination method and device, equipment and product

Abstract

The invention relates to a model training method, an image category determination method and device, electronic equipment and a computer program product, relates to the technical field of multi-modal recognition, and can be applied to a scene for recognizing an article category contained in an image. The method comprises the following steps: acquiring a plurality of image blocks corresponding to a training image, wherein the plurality of image blocks are obtained by performing image preprocessing on the training image; obtaining a pre-configured candidate category label and a pre-constructed loop detection model; and inputting the candidate category labels and the plurality of image blocks into a loop detection model, training the loop detection model until a current prediction category output by the loop detection model at the current moment is the same as a previous prediction category output by the loop detection model at the previous moment, and obtaining an image category determination model. The image blocks of the training image are adopted as model input, so that the model can combine the global features and the local features, the problem that the edge object detection effect is poor is solved, and the edge detection effect of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of multimodal recognition, and in particular, to a model training method, an image category determination method, a model training device, an image category determination device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Multimodal Large Language Model (MLLM) refers to a model that combines multimodal information such as text, images, videos, and audio for training. Multimodal large models were proposed in the context of the development of visual inspection tasks to overcome the limitations of traditional visual models in understanding and interpreting complex scenes. These models achieve a deeper and more comprehensive understanding of the environment and content by integrating multiple types of data modalities, such as images, text, videos, and audio.

[0003] The processing steps of the traditional visual model include inputting the image into the image detection network, and the image detection network analyzes the input image to obtain the corresponding output result.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a model training method, an image category determination method, a model training device, an image category determination device, an electronic device, a computer-readable storage medium, and a computer program product, thereby at least to a certain extent overcoming the problem that the relevant visual detection model has poor object detection effect at the edge of the image and is prone to false detection of non-existent objects.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0007] According to a first aspect of the present disclosure, a model training method is provided, comprising: obtaining a plurality of image blocks corresponding to a training image, the plurality of image blocks being obtained by performing image preprocessing on the training image; obtaining a preconfigured candidate category label, and a pre-constructed cycle detection model; inputting the candidate category label and the plurality of image blocks into the cycle detection model, and training the cycle detection model until a current predicted category output by the cycle detection model at a current moment is the same as a previous predicted category output at a previous moment, thereby obtaining an image category determination model.

[0008] In an exemplary embodiment of the present disclosure, the obtaining of multiple image blocks corresponding to the training image includes: obtaining a preconfigured image segmentation size and image scaling ratio; segmenting the training image according to the image segmentation size to obtain a first image block; scaling the training image according to the image scaling ratio to obtain a first scaled image; segmenting the first scaled image according to the image segmentation size to obtain a second image block; scaling the first scaled image according to the image scaling ratio to obtain a second scaled image; and splicing the first image block, the second image block and the second scaled image to obtain the multiple image blocks.

[0009] In an exemplary embodiment of the present disclosure, the cycle detection model includes a multimodal pre-trained model and a region candidate network, and the candidate category label and the multiple image blocks are input into the cycle detection model, and the cycle detection model is trained until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, thereby obtaining an image category determination model, including: inputting the candidate category label and the multiple image blocks into the cycle detection model; processing the candidate category label and the multiple image blocks by the multimodal pre-trained model and the region candidate network to obtain a region of interest corresponding to the training image; generating a key area attention image corresponding to the training image based on the region of interest; and performing category prediction processing on the key area attention image by the cycle detection model until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, thereby obtaining the image category determination model.

[0010] In an exemplary embodiment of the present disclosure, generating a key area attention image corresponding to the training image based on the region of interest includes: determining the region position coordinates of the region of interest, and determining the region center point of the region of interest according to the region position coordinates; obtaining a pre-configured image selection size, and determining the key area of ​​attention of the training image based on the region center point and the image selection size; adding visual prompt information to the key area of ​​attention to generate the key area attention image.

[0011] In an exemplary embodiment of the present disclosure, generating a key area attention image corresponding to the training image based on the region of interest includes: obtaining the region position coordinates of the region of interest; determining the region to be blurred of the training image based on the region position coordinates; when there are multiple regions to be blurred, performing piecewise Gaussian blur processing on the regions to be blurred using different blur standard deviations to obtain multiple different blurred image regions; generating the key area attention image based on the region of interest and at least one of the blurred image regions.

[0012] In an exemplary embodiment of the present disclosure, generating a key area attention image corresponding to the training image based on the region of interest includes: obtaining the region position coordinates of the region of interest; determining the area to be processed of the training image based on the region position coordinates; graying the area to be processed to obtain a grayscale image area; generating the key area attention image based on the region of interest and the grayscale image area.

[0013] According to a second aspect of the present disclosure, there is provided an image category determination method, comprising: obtaining an image to be identified, and candidate category information corresponding to the image to be identified; obtaining a pre-trained image category determination model, wherein the image category determination model is trained based on the above-mentioned model training method; and inputting the image to be identified and the candidate category information into the image category determination model to obtain a category detection result corresponding to the image to be identified.

[0014] According to a third aspect of the present disclosure, a model training device is provided, comprising: an image acquisition module, used to acquire multiple image blocks corresponding to a training image, wherein the multiple image blocks are obtained by performing image preprocessing on the training image; an initial model construction module, used to acquire pre-configured candidate category labels and a pre-constructed cycle detection model; and a model training module, used to input the candidate category labels and the multiple image blocks into the cycle detection model, and train the cycle detection model until a current predicted category output by the cycle detection model at a current moment is the same as a previous predicted category output at a previous moment, thereby obtaining an image category determination model.

[0015] In an exemplary embodiment of the present disclosure, the image acquisition module includes an image acquisition unit, which is used to: obtain a pre-configured image segmentation size and an image scaling ratio; segment the training image according to the image segmentation size to obtain a first image block; scale the training image according to the image scaling ratio to obtain a first scaled image; segment the first scaled image according to the image segmentation size to obtain a second image block; scale the first scaled image according to the image scaling ratio to obtain a second scaled image; and splice the first image block, the second image block and the second scaled image to obtain the multiple image blocks.

[0016] In an exemplary embodiment of the present disclosure, the cycle detection model includes a multimodal pre-trained model and a region candidate network, and the model training module includes a model training unit, which is used to: input the candidate category label and the multiple image blocks into the cycle detection model; process the candidate category label and the multiple image blocks by the multimodal pre-trained model and the region candidate network to obtain the region of interest corresponding to the training image; generate a key area attention image corresponding to the training image based on the region of interest; and perform category prediction processing on the key area attention image by the cycle detection model until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, thereby obtaining the image category determination model.

[0017] In an exemplary embodiment of the present disclosure, the model training unit includes a first area image generation subunit, which is used to: determine the area position coordinates of the area of ​​interest, and determine the area center point of the area of ​​interest based on the area position coordinates; obtain a pre-configured image selection size, and determine the key focus area of ​​the training image based on the area center point and the image selection size; add visual prompt information to the key focus area to generate the key area attention image.

[0018] In an exemplary embodiment of the present disclosure, the model training unit includes a second regional image generation subunit, which is used to: obtain the regional position coordinates of the region of interest; determine the region to be blurred of the training image based on the regional position coordinates; when the number of the regions to be blurred is multiple, use different blur standard deviations to perform segmented Gaussian blur processing on the regions to be blurred to obtain multiple different blurred image regions; based on the region of interest and at least one of the blurred image regions, generate the key area focus image.

[0019] In an exemplary embodiment of the present disclosure, the model training unit includes a third area image generation subunit, which is used to: obtain the area position coordinates of the area of ​​interest; determine the area to be processed of the training image based on the area position coordinates; grayscale the area to be processed to obtain a grayscale image area; and generate the key area focus image based on the area of ​​interest and the grayscale image area.

[0020] According to a fourth aspect of the present disclosure, there is provided an image category determination device, comprising: an information acquisition module, used to acquire an image to be identified, and candidate category information corresponding to the image to be identified; a model acquisition module, used to acquire a pre-trained image category determination model, wherein the image category determination model is trained based on the above-mentioned model training method; and a category detection module, used to input the image to be identified and the candidate category information into the image category determination model to obtain a category detection result corresponding to the image to be identified.

[0021] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the model training method according to any one of the above items is implemented, or the image category determination method described above is implemented.

[0022] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model training method according to any one of the above items is implemented, or the image category determination method described above is implemented.

[0023] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any one of the above-described model training methods, or implements the above-described image category determination method.

[0024] The technical solution provided by the present disclosure may have the following beneficial effects:

[0025] The model training method in the exemplary embodiment of the present disclosure, on the one hand, uses a loop detection model for multimodal recognition, which can optimize the correspondence between text queries and images, exclude categories that do not exist in the image, and achieve category detection with zero-sample learning. On the other hand, using image blocks of training images as model input allows the model to combine global features with local features, improve the problem of poor edge object detection, and improve the edge detection effect of the model.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0028] Figure 1 The framework diagram of the multimodal detection system used in the related art is schematically shown;

[0029] Figure 2 A flowchart of a model training method according to an exemplary embodiment of the present disclosure is schematically shown;

[0030] Figure 3 A diagram schematically shows a training process of a cycle detection model according to an exemplary embodiment of the present disclosure;

[0031] Figure 4 A process diagram of determining an image prediction category using a cycle detection model according to an exemplary embodiment of the present disclosure is schematically shown;

[0032] Figure 5 A schematic diagram schematically shows a diagram of obtaining an image pyramid after preprocessing an image according to an exemplary embodiment of the present disclosure;

[0033] Figure 6 A schematic diagram schematically shows a schematic diagram of performing piecewise Gaussian blurring on a region other than a region of interest according to an exemplary embodiment of the present disclosure;

[0034] Figure 7 A flowchart of an image category determination method according to an exemplary embodiment of the present disclosure is schematically shown;

[0035] Figure 8 A block diagram of a model training device according to an exemplary embodiment of the present disclosure is schematically shown;

[0036] Fig. 9 A block diagram schematically shows an apparatus for determining an image category according to an exemplary embodiment of the present disclosure;

[0037] Fig.10 A schematic diagram schematically illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure;

[0038] Fig.11 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0039] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0040] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0042] Existing multimodal detection methods often combine image and text information. Figure 1 , Figure 1 The framework diagram of the multimodal detection system used in the related art is schematically shown. Figure 1 It can be seen that the currently used multimodal detection system mainly completes the following functions: (1) inputting the image into the image encoder to extract the image features; (2) inputting the text into the text encoder to extract the text features; (3) fusing the text features with the image features to obtain the category detection results of the image.

[0043] However, the above multimodal detection system is prone to losing pixel information at the edge of the image when extracting features from the image, so it has a better detection effect on objects at the center of the image, but a poorer detection effect on objects at the edge of the image.

[0044] Based on this, in this example embodiment, a model training method is first provided. The model training method of the present disclosure can be implemented using a server, or the method described in the present disclosure can be implemented using a terminal device, wherein the terminal described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptops, PDAs, personal digital assistants (PDAs), and fixed terminals such as desktop computers. Figure 2 The following is a schematic diagram of the model training method flow according to some embodiments of the present disclosure. Figure 2 , the model training method may include the following steps:

[0045] Step S210, obtaining a plurality of image blocks corresponding to the training image, wherein the plurality of image blocks are obtained by performing image preprocessing on the training image.

[0046] According to some exemplary embodiments of the present disclosure, the training image may be an image used for training a model, such as the training image of the present disclosure may be used for training an image category determination model. The image block may be a plurality of image blocks obtained after image preprocessing the training image. Image preprocessing may be a process of performing operations such as scaling or segmenting the training image to obtain a plurality of image blocks.

[0047] refer to Figure 3 , Figure 3 The training process diagram of the cycle detection model according to the exemplary embodiment of the present disclosure is schematically shown. A training image for model training is obtained, and image preprocessing is performed on the training image, such as image segmentation processing, image scaling processing, and scaling and segmentation combined processing, etc., to obtain multiple image blocks corresponding to the training image. The multiple image blocks of the obtained training image are used as the model input of the training model, so that the model can extract features from different image areas during the training process to obtain the category recognition result of the training image.

[0048] Step S220, obtaining pre-configured candidate category labels and a pre-built cycle detection model.

[0049] According to some exemplary embodiments of the present disclosure, the candidate category labels may be the category labels of items that may be included in the training image. The loop detection model may be a network model that performs loop detection on the candidate category labels and the image blocks to determine the category detection results of the training image. For example, the loop detection model may include a network model of a recurrent neural network (RNN) structure.

[0050] Continue to refer Figure 3, obtain the candidate category labels corresponding to the training image, which can be category labels that may match the training image determined based on the image text dataset. After obtaining the candidate category labels of the training image, the candidate category labels are subjected to text encoding, and multiple image blocks of the training image and the text-encoded candidate category labels are used as model input data to train the pre-built cycle detection model.

[0051] Step S230, input the candidate category labels and multiple image blocks into the loop detection model, train the loop detection model, and obtain the image category determination model until the current predicted category output by the loop detection model at the current moment is the same as the previous predicted category output at the previous moment.

[0052] According to some exemplary embodiments of the present disclosure, the cycle detection model performs multiple rounds of model training, and the current moment may be the moment corresponding to when the cycle detection model outputs the predicted category in the current round, and the current moment may be recorded as moment t; the previous moment may be the moment corresponding to when the cycle detection model outputs the predicted category in the previous round, and the previous moment may be recorded as moment t-1. The current predicted category may be the category prediction result output by the cycle detection model in the current round (such as moment t). The previous predicted category may be the category prediction result output by the cycle detection model in the previous round (such as moment t-1). The image category determination model may be a model obtained after model training, which can be used to detect the category of items contained in the image and output the corresponding category recognition result.

[0053] Continue to refer Figure 3 , multiple image blocks of the training image and the candidate category labels after text encoding are input into the cycle detection model, and the cycle detection model outputs the detected valid categories and excludes the invalid categories. The cycle detection model can output one or more intermediate prediction results at different times of the round. For example, in the first round (time 1), the corresponding intermediate prediction result h is output. 1 , in the second round (time 2), the corresponding intermediate prediction result h is output 2 , output the corresponding intermediate prediction result h in the t-1th round (time t-1) t-1 , output the corresponding intermediate prediction result h at the tth round (time t) t wait.

[0054] During the model training process, when the category detected in the tth round (time t) of model training is consistent with the category detected in the t-1th round (time t-1), the cycle is stopped, and the cyclic detection model trained in the tth round (time t) can be used as the image category determination model; when the category detected at time t is inconsistent with the category detected at time t-1, the model training continues until the model training stop condition is met (that is, the predicted category output in the current round is consistent with the predicted category output in the previous round). The cyclic detection network can optimize the correspondence between text queries and images and improve the model detection effect.

[0055] It should be noted that the cycle detection model in the present disclosure can perform multiple rounds of model training, and the current time t and the previous time t-1 are only used to distinguish different training rounds corresponding to the cycle detection model, such as the time t corresponding to the current round and the time t-1 corresponding to the previous round.

[0056] According to the model training method in this example embodiment, on the one hand, the use of a cyclic detection model for multimodal recognition can optimize the correspondence between text queries and images, exclude categories that do not exist in the image, and achieve zero-sample learning category detection. On the other hand, using image blocks of training images as model input allows the model to combine global features with local features, improve the problem of poor edge object detection, and improve the edge detection effect of the model.

[0057] The model training method in this example embodiment will be further described below.

[0058] In an exemplary embodiment of the present disclosure, for step S210, obtaining multiple image blocks corresponding to the training image includes: obtaining a pre-configured image segmentation size and image scaling ratio; segmenting the training image according to the image segmentation size to obtain a first image block; scaling the training image according to the image scaling ratio to obtain a first scaled image; segmenting the first scaled image according to the image segmentation size to obtain a second image block; scaling the first scaled image according to the image scaling ratio to obtain a second scaled image; and splicing the first image block, the second image block, and the second scaled image to obtain multiple image blocks.

[0059] The image segmentation size may be the size used to segment the image, and the image scaling ratio may be the scale used to scale the image.

[0060] refer to Figure 4 , Figure 4The process diagram of determining the image prediction category using a cyclic detection model according to an exemplary embodiment of the present disclosure is schematically shown. After the training image is acquired, in step S410, the training image is preprocessed to obtain multiple image blocks. For example, the training image is processed using an image pyramid image expression. An image pyramid is a set of sub-images of different resolutions of the same image, which is generated by continuously downsampling the original image, that is, a low-resolution approximate image (small size) is generated from a high-resolution image (large size).

[0061] refer to Figure 5 , Figure 5 A schematic diagram schematically shows an image pyramid obtained after preprocessing an image according to an exemplary embodiment of the present disclosure. Figure 5 The image segmentation size can be configured as 1 / 2, 1 / 4, etc. of the original image size, and the image scaling ratio can be configured as 1 / 2, 1 / 4, etc. of the image. For example, Figure 5 The training image is reduced twice, and the image segmentation size of the training image can be configured to be 1 / 4 of the original image. At this time, the training image is segmented to obtain the first image block, that is, 16 image blocks (patch) are obtained as the first image block.

[0062] Then, the training image is scaled, for example, the image scaling ratio is configured to 1 / 2, and the training image is reduced according to the scaling ratio 1 / 2 to obtain a first scaled image. After obtaining the first scaled image, the first scaled image is segmented according to the image segmentation size 1 / 2 to obtain 4 image blocks as the second image blocks.

[0063] For the first scaled image obtained, the first scaled image can be further scaled according to the image scaling ratio of 1 / 2 to obtain a second scaled image. Then, the first image block, the second image block and the second scaled image are spliced ​​to obtain multiple image blocks. In this embodiment, the original image corresponding to the training image is scaled twice at a ratio of 1 / 2 and cut into 21 patches for splicing. By scaling the image to form a pyramid and cutting it into image blocks, more comprehensive information can be extracted for model training and the edge detection effect can be improved.

[0064] It is easy for those skilled in the art to understand that in some other exemplary embodiments of the present disclosure, other image segmentation sizes and image scaling ratios may be used to segment and scale the training images to obtain the corresponding image pyramid. The present disclosure does not impose any special limitation on the number of image pyramid layers and the specific values ​​of the image segmentation size and the image scaling ratio.

[0065] In an exemplary embodiment of the present disclosure, for step S220, the candidate category label and multiple image blocks are input into a cycle detection model, and the cycle detection model is trained until the current prediction category output by the cycle detection model at the current moment is the same as the previous prediction category output at the previous moment, and an image category determination model is obtained, including: inputting the candidate category label and multiple image blocks into the cycle detection model; processing the candidate category label and multiple image blocks by a multimodal pre-trained model and a regional candidate network to obtain a region of interest corresponding to the training image; based on the region of interest, generating a key area attention image corresponding to the training image; and performing category prediction processing on the key area attention image by the cycle detection model until the current prediction category output by the cycle detection model at the current moment is the same as the previous prediction category output at the previous moment, and an image category determination model is obtained.

[0066] Among them, the multimodal pre-training model can be a network model that takes data of multiple modalities (such as images, text, video, audio, etc.) as input. For example, the multimodal pre-training model can be a CLIP model. CLIP (Contrastive Language-Image Pre-training) is a multimodal (text and image) pre-training model. The Region Proposal Network (RPN) can be a network for extracting candidate boxes, which can be used to extract candidate boxes (suggestion boxes). The region of interest can be an area to be processed that is outlined from the training image in the form of a square, circle, ellipse, irregular polygon, etc. The key area of ​​attention image can be an image obtained after adding visual prompt information to the region of interest in the sample image.

[0067] Continue to refer Figure 4 After obtaining multiple image blocks corresponding to the training image, the preprocessed image blocks and candidate category labels are input into the cycle detection model. The cycle detection model may include a multimodal pre-training model (CLIP model) and a region candidate network (RPN network). The CLIP model and the RPN network process the candidate category labels and multiple image blocks to determine the region of interest corresponding to the training image.

[0068] For example, the CLIP model can process multiple input image blocks and text information, and the RPN network can output multiple proposal boxes (proposals), and the RPN network determines the region of interest in the training image based on the multiple proposal boxes. The specific process of the CLIP model and the RPN network is the same as the related scheme, and this disclosure will not repeat it.

[0069] After determining the region of interest of the training image, visual cue information is added to the region of interest to generate a key region attention image corresponding to the training image. Then, the key region attention image is preprocessed and used as the input of the cycle detection model for category prediction processing until the current prediction category output by the cycle detection model at the current moment is the same as the previous prediction category output at the previous moment, and then the image category determination model is obtained. By adding visual cue information to the region of interest, the model's attention to specific areas of the image can be enhanced, thereby improving the accuracy of the model's recognition results.

[0070] In an exemplary embodiment of the present disclosure, a key area attention image corresponding to a training image is generated based on a region of interest, including: determining the region position coordinates of the region of interest, and determining the region center point of the region of interest according to the region position coordinates; obtaining a pre-configured image selection size, and determining the key area of ​​attention of the training image based on the region center point and the image selection size; adding visual prompt information to the key area of ​​attention to generate a key area attention image.

[0071] The region position coordinates may be the position coordinates of the region of interest in the training image. The region center point may be the center point corresponding to the region of interest. The image selection size may be the selection size used to select the focus region from the training image. The focus region may be the region in the training image to which visual prompt information needs to be added. The visual prompt information may be identification information used to prompt the model that the information in this region needs to be focused on.

[0072] The training image is input into the RPN network, which can output the suggestion box corresponding to the training image and determine the region of interest y of the training image based on the suggestion box. t If a rectangular frame is used to enclose the region of interest, the RPN network can output the position coordinates of the four points of the rectangular frame. The region center point of the region of interest can be determined based on the above region position coordinates. For example, the region center point corresponding to the rectangular frame can be calculated by half of the absolute value of the difference between the position coordinate value of the upper left corner point and the position coordinate value of the lower right corner point. The horizontal axis coordinate value of the region center point can be obtained according to the above calculation method.

[0073] After obtaining the center point of the region, obtain the pre-configured image selection size, for example, the image selection size can be configured as half of the width of the training image. With the center point of the region as the center and the image selection size as the radius, a key focus area is circled; and visual prompt information is added to the key focus area for key marking (for example, a red circle is used to mark the area), and the training image with the visual prompt information is used as the key focus area image.

[0074] Continue to refer Figure 4After obtaining the focus area image, the focus area image is preprocessed and cut into image blocks again. At this time, the preprocessed image blocks contain visual cue information. The preprocessed image blocks are input into the CLIP model and classifier, and the corresponding prediction category results are output through the loop detection model. When h t and h t-1 When they are consistent, the loop stops and the image category determination model is obtained. By adding specific annotations to the image to circle the focus area, the model can enhance its attention to specific areas of the image.

[0075] For example, after obtaining multiple image blocks after preprocessing, the multiple image blocks are input into the CLIP model and the RPN network; at the same time, the initial candidate label information h 0 Input to the CLIP+RPN network, such as h 0 It can be [sofa, desk, people, car]. The CLIP+RPN network outputs a suggestion box and determines the region of interest. The region of interest and the suggestion box can be the same or different. For example, a suggestion box can be directly used as the region of interest.

[0076] After generating the focus area image based on the region of interest, the focus area image is preprocessed and cut into image blocks again. The obtained multiple image blocks are input into the CLIP+ classifier to obtain the predicted label h. 1 , such as h 1 It can be [sofa,desk], h 1 With the last tag information h 0 If not consistent, 1 The image block is input into the CLIP+RPN network again, and the above steps of outputting the suggestion box and determining the predicted label based on the generated focus area attention image are continued in a loop to obtain h 2 , such as h 2 It can be [sofa,desk], h 2 With h 1 If they are not consistent, stop learning.

[0077] It is easy for those skilled in the art to understand that in other exemplary embodiments of the present disclosure, selection frames of other shapes may be used to encircle the key focus area, such as a rectangle, an ellipse, an irregular shape, etc. The present disclosure does not impose any special limitation on the shape of the key focus area.

[0078] In an exemplary embodiment of the present disclosure, based on the region of interest, a key area attention image corresponding to the training image is generated, including: obtaining the region position coordinates of the region of interest; based on the region position coordinates, determining the region to be blurred of the training image; when the number of regions to be blurred is multiple, using different blur standard deviations to perform piecewise Gaussian blur processing on the region to be blurred to obtain multiple different blurred image regions; based on the region of interest and at least one blurred image region, a key area attention image is generated.

[0079] The area to be blurred may be an image area to be blurred. The blur standard deviation may be a parameter value used for Gaussian blurring of the area to be blurred. The blurred image area may be an area in the training image where a low-resolution image obtained after blurring the area to be blurred is located.

[0080] The region of interest y of the training image determined by the RPN network t , containing the region of interest y t The region position coordinates of the region of interest can be used to determine the region to be blurred in the training image according to the region position coordinates of the region of interest. For example, all regions outside the region of interest in the training image are used as the region to be blurred. The region to be blurred can be blurred as a whole region, or the region to be blurred can be blurred in sections.

[0081] When the number of regions to be blurred is one, the same blur parameter is used to blur the entire region. When the number of regions to be blurred is multiple, different blur standard deviations can be used to perform segmented Gaussian blur processing on the regions to be blurred. For example, according to the region of interest y t Different standard deviation values ​​(σ 1 , σ 2 ,…),refer to Figure 6 , Figure 6 The schematic diagram of performing segmented Gaussian blurring on regions other than the region of interest according to an exemplary embodiment of the present disclosure is schematically shown. Gaussian blurring is performed on different regions to be blurred using the following formula 1 to obtain corresponding blurred image regions, such as the first blurred region, the second blurred region, etc.

[0082]

[0083] The larger the σ value, the higher the blur of the image, and the smaller the σ value, the less blur the image. For example, the pixels closer to the region of interest are more important and need to be less blurred, while the pixels closer to the edge are less important and can be more blurred.

[0084] After obtaining multiple different blurred image regions, the region of interest in the training image and one or more blurred image regions are used as the focus region attention image. Then, the focus region attention image is preprocessed again and cut into multiple image blocks. The preprocessed image blocks are input into the CLIP model and classifier, and the corresponding prediction category results are output through the loop detection model. When h t and h t-1 When they are consistent, the loop stops and the image category determination model is obtained. By performing segmented Gaussian blur processing on the original image to determine the focus area, the model's attention to specific areas of the image can be enhanced.

[0085] For example, after obtaining multiple image blocks after preprocessing, the multiple image blocks are input into the CLIP model and the RPN network; at the same time, the initial candidate label information h 0 Input to the CLIP+RPN network, such as h 0 It can be [sofa, desk,], the CLIP+RPN network outputs the suggestion box and determines the region of interest.

[0086] According to the region of interest, the background part of the original training image is blurred to provide visual cues, and the region of interest retains the original pixels; outside the region of interest, a Gaussian blur with a standard deviation of 5 is performed within a 2-fold area to obtain the first blurred area; outside the 2-fold area of ​​the region of interest, a Gaussian blur with a standard deviation of 10 is performed within a 3-fold area to obtain the second blurred area; until the edge of the picture, multiple different blurred image areas are obtained.

[0087] Generate a focus image based on the region of interest and the blurred image region, and cut it into image blocks again. Input multiple image blocks into the CLIP+ classifier to obtain the predicted label h 1 , such as h 1 It can be [sofa,desk], h 1 With the last tag information h 0 If they are not consistent, stop learning.

[0088] In an exemplary embodiment of the present disclosure, a key area attention image corresponding to a training image is generated based on a region of interest, including: obtaining the region position coordinates of the region of interest; determining a region to be processed of the training image based on the region position coordinates; graying the region to be processed to obtain a grayscale image region; and generating a key area attention image based on the region of interest and the grayscale image region.

[0089] The area to be processed may be an image area to be processed. Grayscale processing is a process of converting a color image into a grayscale image. The grayscale image area may be an image area that has been processed by grayscale.

[0090] The region of interest y of the training image determined by the RPN network t After obtaining the region position coordinates of the region of interest, the region to be processed of the training image can be determined according to the region position coordinates of the region of interest, such as taking the regions outside the region of interest in the training image as the region to be processed. Then, the region to be processed is grayed to obtain a gray image region; based on the region of interest and the gray image region, a focus image of the key region is generated. By graying the regions outside the region of interest in the original training image, the specific regions that need to be focused on can be highlighted, and the model's attention to the specific regions of the image can be enhanced.

[0091] It is easy for those skilled in the art to understand that in some other exemplary embodiments of the present disclosure, other methods can also be used to determine the key focus areas in the training image, such as adding masks to areas outside the area of ​​interest, using softening edges, etc., to achieve the effect of enhancing the model's focus on specific areas of the image, and the present disclosure does not make any special limitations on this.

[0092] It should be noted that the terms “first”, “second”, etc. used in the present disclosure are only used to distinguish different scaled images and different image blocks, and should not impose any limitations on the present disclosure.

[0093] In summary, the model training method disclosed in the present invention obtains multiple image blocks corresponding to the training image, and the multiple image blocks are obtained by image preprocessing the training image; obtains pre-configured candidate category labels and a pre-constructed cycle detection model; inputs the candidate category labels and the multiple image blocks into the cycle detection model, and trains the cycle detection model until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, thereby obtaining an image category determination model. On the one hand, the use of the cycle detection model for multimodal recognition can optimize the correspondence between text queries and images, exclude categories that do not exist in the image, and leave valid categories for detection, without the need to learn and re-detect non-existent categories, thereby achieving zero-sample learning category detection. On the other hand, the image blocks of the training image are used as model input, so that the model can combine global features with local features, improve the problem of poor edge object detection effect, and improve the edge detection effect of the model. On the other hand, by performing regional annotation or segmented Gaussian blurring on the training image, the model's attention to specific areas of the image can be enhanced.

[0094] Secondly, the present disclosure also provides a method for determining an image category, which can be implemented by using a server or by using a terminal device. Figure 7The following is a schematic diagram schematically showing the image category determination method flow according to some embodiments of the present disclosure. Figure 7 , the image category determination method may include the following steps:

[0095] Step S710: obtaining an image to be identified and candidate category information corresponding to the image to be identified.

[0096] According to some exemplary embodiments of the present disclosure, the image to be identified may be an image to be processed for category identification. The candidate category information may be a candidate category label that the image to be identified may correspond to.

[0097] The image that needs to be identified as the item category is used as the image to be identified, and the candidate category information corresponding to the image is obtained. The candidate category information can be determined by obtaining user input data, such as the candidate category information can be a category text label manually input by the user; it can also be determined by other methods, such as according to the user's selection operation. Specifically, the optional category label is recommended to the user, and the item category label selected by the user is used as the candidate category information; the candidate category information can also be automatically generated. The candidate category information can be expressed in Chinese or English. For example, the candidate category information can include "[sofa, desk, people, car]" and so on.

[0098] Step S720, obtaining a pre-trained image category determination model, where the image category determination model is trained based on a model training method.

[0099] According to some exemplary embodiments of the present disclosure, an image category determination model trained according to the above-mentioned model training method is obtained. The model can detect the category of objects contained in the image to be identified, understand objects that do not exist in the image, avoid false detection, and realize a zero-sample learning detection system.

[0100] Step S730: input the image to be identified and the candidate category information into an image category determination model to obtain a category detection result corresponding to the image to be identified.

[0101] According to some exemplary embodiments of the present disclosure, the image to be identified and the candidate category information are input into the image category determination model to obtain the category detection result corresponding to the image to be identified. For example, after the model recognition processing, the category detection result obtained by the present disclosure is "[sofa, desk]", and the image to be identified does not contain the item categories of "people" and "car".

[0102] According to the image category determination method in this example embodiment, on the one hand, the image category determination model is implemented using a loop detection model, which can optimize the correspondence between text queries and images, and the model can understand objects that do not exist in the image, avoid false detection, and realize a zero-sample learning detection system. On the other hand, the model can extract more comprehensive information from the image to be identified and can enhance attention to specific areas of the image.

[0103] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0104] In addition, in this exemplary embodiment, a model training device is also provided. Figure 8 The model training device 800 may include: an image acquisition module 810, an initial model construction module 820 and a model training module 830.

[0105] Specifically, the image acquisition module 810 is used to obtain multiple image blocks corresponding to the training image, and the multiple image blocks are obtained by performing image preprocessing on the training image; the initial model construction module 820 is used to obtain pre-configured candidate category labels and a pre-constructed cycle detection model; the model training module 830 is used to input the candidate category labels and the multiple image blocks into the cycle detection model, and train the cycle detection model until the current prediction category output by the cycle detection model at the current moment is the same as the previous prediction category output at the previous moment, so as to obtain the image category determination model.

[0106] In an exemplary embodiment of the present disclosure, the image acquisition module 810 includes an image acquisition unit, which is used to: obtain a pre-configured image segmentation size and an image scaling ratio; segment a training image according to the image segmentation size to obtain a first image block; scale the training image according to the image scaling ratio to obtain a first scaled image; segment the first scaled image according to the image segmentation size to obtain a second image block; scale the first scaled image according to the image scaling ratio to obtain a second scaled image; and splice the first image block, the second image block, and the second scaled image to obtain multiple image blocks.

[0107] In an exemplary embodiment of the present disclosure, the cycle detection model includes a multimodal pre-trained model and a region candidate network, and the model training module 830 includes a model training unit, which is used to: input candidate category labels and multiple image blocks into the cycle detection model; process the candidate category labels and multiple image blocks by the multimodal pre-trained model and the region candidate network to obtain a region of interest corresponding to the training image; generate a key area attention image corresponding to the training image based on the region of interest; and perform category prediction processing on the key area attention image by the cycle detection model until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, thereby obtaining an image category determination model.

[0108] In an exemplary embodiment of the present disclosure, the model training unit includes a first regional image generation subunit, which is used to: determine the regional position coordinates of the region of interest, and determine the regional center point of the region of interest based on the regional position coordinates; obtain a pre-configured image selection size, and determine the key focus area of ​​the training image based on the regional center point and the image selection size; add visual prompt information to the key focus area to generate a key area focus image.

[0109] In an exemplary embodiment of the present disclosure, the model training unit includes a second regional image generation subunit, which is used to: obtain the regional position coordinates of the region of interest; determine the region to be blurred of the training image based on the regional position coordinates; when the number of regions to be blurred is multiple, use different blur standard deviations to perform piecewise Gaussian blur processing on the region to be blurred to obtain multiple different blurred image regions; based on the region of interest and at least one blurred image region, generate a key area focus image.

[0110] In an exemplary embodiment of the present disclosure, the model training unit includes a third area image generation subunit, which is used to: obtain the area position coordinates of the area of ​​interest; determine the area to be processed of the training image based on the area position coordinates; grayscale the area to be processed to obtain a grayscale image area; and generate a key area focus image based on the area of ​​interest and the grayscale image area.

[0111] In addition, in this exemplary embodiment, an image category determination device is also provided. Fig. 9 The image category determination device 900 may include: an information acquisition module 910, a model acquisition module 920 and a category detection module 930.

[0112] Specifically, the information acquisition module 910 is used to obtain the image to be identified and the candidate category information corresponding to the image to be identified; the model acquisition module 920 is used to obtain a pre-trained image category determination model, which is trained based on the above-mentioned model training method; the category detection module 930 is used to input the image to be identified and the candidate category information into the image category determination model to obtain the category detection result corresponding to the image to be identified.

[0113] The specific details of the virtual modules of the above-mentioned model training devices and image category determination devices have been described in detail in the corresponding model training methods and image category determination methods. The undisclosed details can be found in the implementation method content of the method part, and will not be repeated here.

[0114] It should be noted that although several modules or units of the model training device and the image category determination device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0115] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any one of the above-mentioned model training methods or the above-mentioned image category determination method.

[0116] In one embodiment, the computer program product may be a tangible product containing the computer program, such as a computer-readable storage medium storing the computer program. Fig.10 , Fig.10 The schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid state drive (SSD), etc. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), etc.

[0117] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0118] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0119] The computer program can be carried or transmitted by electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device can convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device can execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned model training method and image category determination method.

[0120] The exemplary embodiments of the present disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0121] like Fig.11 As shown, the electronic device 1100 may include: a processor 1110 , a memory 1120 , a bus 1130 , an I / O (input / output) interface 1140 , a network adapter 1150 , and a display 1160 .

[0122] The memory 1120 may include a volatile memory, such as a RAM 1121, a cache unit 1122, and may also include a non-volatile memory, such as a ROM 1123. The memory 1120 may also include one or more program modules 1124, such program modules 1124 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 1124 may include each module in the above-mentioned device.

[0123] Processor 1110 may include one or more processing units. For example, processor 1110 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0124] The processor 1110 may be used to execute executable instructions stored in the memory 1120, such as executing any one of the model training methods described above, or implementing the above-mentioned image category determination method.

[0125] The bus 1130 is used to realize the connection between different components of the electronic device 1100, and may include a data bus, an address bus, and a control bus.

[0126] The electronic device 1100 can communicate with one or more external devices 1500 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 1140 .

[0127] The electronic device 1100 can communicate with one or more networks through the network adapter 1150. For example, the network adapter 1150 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 1150 can communicate with other modules of the electronic device 1100 through the bus 1130.

[0128] The electronic device 1100 may display a graphical user interface, such as an image recognition result interface, through the display 1160. The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., a router, a modem, etc.).

[0129] although Fig.11 Not shown, other hardware and / or software modules may also be provided in the electronic device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0130] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0131] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A model training method, characterized in that: include: Acquire a plurality of image blocks corresponding to a training image, wherein the plurality of image blocks are obtained by performing image preprocessing on the training image; Get pre-configured candidate class labels and pre-built cycle detection models; The candidate category labels and the multiple image blocks are input into the cycle detection model, and the cycle detection model is trained until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, thereby obtaining an image category determination model.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of image blocks corresponding to the training image includes: Get the pre-configured image segmentation size and image scaling ratio; Segmenting the training image according to the image segmentation size to obtain a first image block; Scaling the training image according to the image scaling ratio to obtain a first scaled image; Segmenting the first scaled image according to the image segmentation size to obtain second image blocks; Scaling the first scaled image according to the image scaling ratio to obtain a second scaled image; The first image block, the second image block and the second zoomed image are spliced ​​to obtain the multiple image blocks.

3. The method according to claim 1, characterized in that The cycle detection model includes a multimodal pre-training model and a region candidate network. The step of inputting the candidate category labels and the plurality of image blocks into the cycle detection model and training the cycle detection model until a current predicted category output by the cycle detection model at a current moment is the same as a previous predicted category output at a previous moment, thereby obtaining an image category determination model comprises: Inputting the candidate category labels and the plurality of image blocks into the cycle detection model; The candidate category labels and the multiple image blocks are processed by the multimodal pre-trained model and the region candidate network to obtain a region of interest corresponding to the training image; Based on the region of interest, generating a focus area image corresponding to the training image; The cycle detection model performs category prediction processing on the key area focus image until the current prediction category output by the cycle detection model at the current moment is the same as the previous prediction category output at the previous moment, thereby obtaining the image category determination model.

4. The method according to claim 3, characterized in that The step of generating a focus area image corresponding to the training image based on the region of interest includes: Determine the region position coordinates of the region of interest, and determine the region center point of the region of interest according to the region position coordinates; Obtaining a pre-configured image selection size, and determining a focus area of ​​the training image based on the center point of the area and the image selection size; Visual prompt information is added to the key focus area to generate the key focus area image.

5. The method according to claim 3, characterized in that: The step of generating a focus area image corresponding to the training image based on the region of interest includes: Obtaining the area position coordinates of the area of ​​interest; Based on the region position coordinates, determining the region to be blurred of the training image; When there are multiple regions to be blurred, using different blur standard deviations to perform segmented Gaussian blur processing on the regions to be blurred to obtain multiple different blurred image regions; The key area attention image is generated based on the region of interest and at least one of the blurred image regions.

6. The method according to claim 3, characterized in that The step of generating a focus area image corresponding to the training image based on the region of interest includes: Obtaining the area position coordinates of the area of ​​interest; Based on the region position coordinates, determining the region to be processed of the training image; Performing grayscale processing on the area to be processed to obtain a grayscale image area; The key area attention image is generated based on the region of interest and the grayscale image region.

7. A method for determining an image category, characterized in that: include: Acquire an image to be identified and candidate category information corresponding to the image to be identified; Obtain a pre-trained image category determination model, wherein the image category determination model is trained based on the model training method according to any one of claims 1 to 6; The image to be identified and the candidate category information are input into the image category determination model to obtain a category detection result corresponding to the image to be identified.

8. A model training device, characterized in that: include: An image acquisition module, used to acquire a plurality of image blocks corresponding to a training image, wherein the plurality of image blocks are obtained by performing image preprocessing on the training image; An initial model building module that obtains pre-configured candidate category labels and a pre-built cycle detection model; A model training module is used to input the candidate category labels and the multiple image blocks into the cycle detection model, and train the cycle detection model until the current predicted category output by the cycle detection model at the current moment is the same as the previous predicted category output at the previous moment, so as to obtain an image category determination model.

9. An image category determination device, characterized in that: include: An information acquisition module, used to acquire an image to be identified and candidate category information corresponding to the image to be identified; A model acquisition module, used to acquire a pre-trained image category determination model, wherein the image category determination model is trained based on the model training method according to any one of claims 1 to 6; The category detection module is used to input the image to be identified and the candidate category information into the image category determination model to obtain a category detection result corresponding to the image to be identified.

10. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the model training method as described in any one of claims 1 to 6, or implement the image category determination method as described in claim 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6, or implements the image category determination method as described in claim 7.

Images