Training method and device of image recognition model

Abstract

The embodiment of the invention discloses a training method and device for an image recognition model, and the method can provide a rich and precise data basis for the subsequent processing through obtaining a target image training data set containing a spatial annotation information sample image set. Spatial perception feature extraction is performed on a sample image set, and a generated multi-level convolution feature set covers feature expressions of different perception granularities, so that various feature details of the image can be comprehensively captured; the feature set is input into an adaptive loss function optimization model, and a cross-level association loss function is generated in combination with a dynamic feature fusion mechanism, so that association among different levels of features can be effectively mined, and more targeted guidance is provided for model optimization; and on the basis of iterative optimization of the initial image recognition model, the trained image recognition model can efficiently carry out end-to-end spatial positioning and semantic analysis on the to-be-recognized target of the input image, and the accuracy and comprehensiveness of image recognition are remarkably improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular to a training method and device of an image recognition model. BACKGROUND

[0002] In the field of image recognition, with the continuous development of artificial intelligence technology, image recognition has been widely applied in many fields. Traditional image recognition technology is committed to simple classification and recognition of targets in images, mainly focusing on surface feature extraction of images, training through fixed model structure and loss function to realize judgment of object categories in images.

[0003] However, the existing image recognition technology has obvious defects. On the one hand, the spatial position information of the target in the image is not sufficiently mined, and it is difficult to accurately locate the specific position of the target in the image, which cannot meet the demand in some scenes with high requirements for spatial positioning. On the other hand, in semantic analysis, due to the lack of mining of effective association between different hierarchical features, the understanding of image content is one-sided and cannot deeply analyze the semantic information of the target in the image. SUMMARY

[0004] Embodiments of the present application provide a training method and device of an image recognition model, which is used to efficiently perform end-to-end spatial positioning and semantic analysis on the to-be-recognized target of the input image by the trained image recognition model, and significantly improves the accuracy and comprehensiveness of image recognition.

[0005] In a first aspect, the present application provides a training method of an image recognition model, applied to a training device of an image recognition model, the method comprising: obtaining a target image training data set, the target image training data set containing a sample image set with spatial annotation information; performing a spatial perception feature extraction operation on the sample image set to generate a multi-level convolution feature set of each sample image, the multi-level convolution feature set containing feature expressions of different perception granularities; inputting the multi-level convolution feature set into a preset adaptive loss function optimization model, and generating a cross-level association loss function in combination with a dynamic feature fusion mechanism; iteratively optimizing an initial image recognition model according to the cross-level association loss function to generate a trained image recognition model; the trained image recognition model is used to perform end-to-end spatial positioning and semantic analysis operations on the to-be-recognized target of the input image.

[0006] In a second aspect, the present application provides a training device of an image recognition model, comprising: a processor; a storage device having a computer program stored thereon, when the computer program is executed by the processor, the processor implements any of the training methods of the image recognition model.

[0007] The embodiment of the present application provides a readable storage medium, and the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize steps of the training method of the image recognition model.

[0008] Therefore, the embodiment of the present application innovatively obtains a sample image set with spatial annotation information as a training data set, and effectively solves the problem that spatial positioning and semantic analysis are not accurate and comprehensive in image recognition through feature extraction, adaptive loss function optimization and a dynamic feature fusion mechanism, and the performance of the image recognition model is improved.

[0009] The embodiment of the present application can provide rich and accurate data basis for subsequent processing by obtaining a target image training data set containing a sample image set with spatial annotation information; the multi-level convolution feature set generated by performing spatial perception feature extraction on the sample image set covers feature expressions of different perception granularities, which is helpful to comprehensively capture various feature details of the image; the feature set is input into an adaptive loss function optimization model, and a cross-level correlation loss function is generated in combination with a dynamic feature fusion mechanism, which can effectively mine the correlation between features of different levels and provide more targeted guidance for model optimization; and based on this, the initial image recognition model is iteratively optimized, so that the trained image recognition model can efficiently perform end-to-end spatial positioning and semantic analysis on an input image to be recognized, and the accuracy and comprehensiveness of image recognition are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A flowchart of the training method of the image recognition model provided by the embodiment of the present application; Figure 2 An exemplary application diagram of the image recognition model provided by the embodiment of the present application; Figure 3 A schematic diagram of the basic structure of the training device of the image recognition model provided by the embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiment of the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0012] Referring to Figure 1 As shown in the figure, the figure is a flowchart of the training method of the image recognition model provided by the embodiment of the present application, and the method can be applied to the training device of the image recognition model. As shown in the figure, Figure 1 As shown in the figure, the method comprises steps 110-140.

[0013] Step 110: obtaining a target image training data set, wherein the target image training data set contains a sample image set with spatial annotation information.

[0014] In the image recognition application scenario of the embodiments of the present application, the target image training data set covers a large number of image resources. Taking natural scene recognition as an example, sample images can be collected from different geographical regions and different time periods, and have diversified scene content.

[0015] Further, each sample image is attached with detailed spatial annotation information, which accurately describes the spatial position and category attribute of each element in the image. For example, in a natural scene image, for animals, plants, buildings and other objects, the spatial annotation information explicitly indicates the specific coordinate range of the objects in the image and the category label to which the objects belong. The annotation work is completed by a professional annotation tool. Annotators carefully annotate each sample image according to strict annotation specifications to ensure the accuracy and consistency of the annotation information, thereby forming a rich and high-quality target image training data set.

[0016] Step 120: performing a spatial perception feature extraction operation on the sample image set to generate a multi-level convolution feature set of each sample image, the multi-level convolution feature set containing feature expressions of different perception granularities.

[0017] This step aims to effectively extract multi-level and multi-granularity feature information from the sample image set to help the model better understand the image semantics.

[0018] In an optional embodiment, the performing a spatial perception feature extraction operation on the sample image set to generate a multi-level convolution feature set of each sample image comprises: step 121: dividing each sample image into a preset number of image region division units, each image region division unit corresponding to a spatial detection region in the target image training data set.

[0019] In specific operation, according to the characteristics of the sample image and the subsequent detection requirements, it is determined to divide each sample image into a preset number of image region division units. The preset number can be determined by comprehensively considering factors such as the size of the image, the distribution of the target to be detected, etc. Each image region division unit corresponds to a corresponding spatial detection region in the target image training data set. For example, in a natural scene image, some image region division units may correspond to regions containing specific animals, and others may correspond to regions containing specific plants or buildings. The above correspondence makes the subsequent feature extraction of each region more targeted, which helps to improve the recognition ability of the model for different targets.

[0020] Step 122: calling a pre-trained convolution feature extraction network to perform a multi-level convolution kernel sliding operation on each image region division unit to generate an initial convolution feature map.

[0021] The pre-trained convolution feature extraction network is trained based on a large amount of image data and has image feature extraction capability. For each image region division unit, a plurality of different parameter convolution kernels are used for multi-level sliding operation. The parameters of the convolution kernel include size, weight, etc. Different size convolution kernels can capture different scale feature information in the image region division unit during the sliding process. For example, a smaller convolution kernel is more suitable for extracting detailed features, and a larger convolution kernel can focus on more macro structural features. During the sliding process, the convolution kernel and the pixels of the image region division unit perform multiplication and accumulation operation at the corresponding position, thereby generating an initial convolution feature map. Through adjustment of the parameter settings of different convolution kernels and the sliding step, a variety of feature information can be extracted, providing a basis for subsequent feature processing.

[0022] Step 123: performing feature channel enhancement processing on the initial convolution feature map, dynamically weighting and fusing the feature channels according to a preset channel attention weight distribution table to generate an enhanced convolution feature map.

[0023] The initial convolution feature map is usually composed of a plurality of feature channels, each channel carrying different types of feature information. The preset channel attention weight distribution table is determined according to the statistical characteristics of the image data and the model requirements, and each feature channel is assigned a corresponding weight. In the feature channel enhancement processing, each feature channel is weighted according to the channel attention weight distribution table. That is, for each channel of the initial convolution feature map, each element in the channel is multiplied by the corresponding weight value. Through the above dynamic weighting and fusion method, the feature information of important channels can be highlighted, and the interference of secondary channels can be suppressed, thereby generating an enhanced convolution feature map and improving the quality and expression ability of the features.

[0024] Step 124: performing cross-modal feature alignment operation on the enhanced convolution feature map and the corresponding image region division unit to generate a multi-level convolution feature set with spatial perception features.

[0025] This step performs cross-modal feature alignment between the enhanced convolution feature map and the original image region division unit, so that the feature information in the convolution feature map and the spatial position information of the image region division unit are better matched and fused. For example, by calculating the similarity, correlation and other indicators between the two, the features are adjusted and aligned. Through the above operation, the generated multi-level convolution feature set not only contains rich feature information, but also has spatial perception ability. The feature expression of different levels can accurately reflect the information of different regions and different granularities in the image, providing more effective data support for subsequent model processing and analysis.

[0026] Step 130: input the multi-level convolutional feature set into a preset adaptive loss function optimization model, and generate a cross-level correlation loss function in combination with a dynamic feature fusion mechanism.

[0027] In this step, the multi-level convolutional feature set can be input as input, and a cross-level correlation loss function that can reflect the difference between model prediction and reality can be generated by using a preset adaptive loss function optimization model and a dynamic feature fusion mechanism.

[0028] In an optional embodiment, the inputting the multi-level convolutional feature set into a preset adaptive loss function optimization model, and generating a cross-level correlation loss function in combination with a dynamic feature fusion mechanism, comprises: step 131: generating a dynamic weight distribution matrix matched in level granularity based on the spatial resolution corresponding to the feature expression of each level in the multi-level convolutional feature set; performing a layer-by-layer weighted aggregation operation on the feature expressions of different levels in the multi-level convolutional feature set through the dynamic weight distribution matrix to generate a cross-level fusion feature vector set.

[0029] First, analyze the spatial resolution corresponding to the feature expression of each level in the multi-level convolutional feature set. Different levels of feature expression have different spatial resolutions due to different convolution operations. According to the differences in these spatial resolutions, a dynamic weight distribution matrix matched thereto is generated, each row of the matrix corresponding to a level and each column corresponding to a different weight parameter. The determination of the weight is based on factors such as spatial resolution and feature importance. For example, a relatively high weight can be assigned to a level with a higher spatial resolution, because these levels contain more detailed information. Through the dynamic weight distribution matrix, a layer-by-layer weighted aggregation operation is performed on the feature expressions of different levels in the multi-level convolutional feature set. That is, the feature expression of each level is multiplied by the corresponding weight in the matrix, and then the results are spliced by level to generate a cross-level fusion feature vector set that integrates the feature information of different levels.

[0030] Step 132: input the cross-level fusion feature vector set into the feature correlation analysis module in the adaptive loss function optimization model, extract the spatial consistency measure value between different level feature vectors; perform a difference compensation correction operation on the cross-level fusion feature vector set according to the spatial consistency measure value to generate a compensated fusion feature vector set; compare the compensated fusion feature vector set with the original feature expressions in the multi-level convolutional feature set layer by layer, and calculate the spatial offset error and semantic classification error of each level feature expression.

[0031] Optionally, the cross-level fusion feature vector set is input into a feature correlation analysis module of the adaptive loss function optimization model, which is used to extract spatial consistency measure values between different level feature vectors. For example, the spatial consistency is measured by calculating the distance, similarity, etc. between the feature vectors. According to these measure values, the cross-level fusion feature vector set is subjected to difference compensation correction operation. If the spatial consistency of the feature vector of a certain level with other levels is poor, it is adjusted accordingly to improve the overall spatial consistency. After correction, a compensated fusion feature vector set is generated. Then, the compensated fusion feature vector set is compared with the original feature expression in the multi-level convolution feature set layer by layer. Through comparison, the spatial offset error and semantic classification error of each level feature expression are calculated. The spatial offset error reflects the deviation degree of the feature in the spatial position, and the semantic classification error reflects the deviation of the feature semantic understanding. The calculation of these errors provides key data for subsequent generation of cross-level correlation loss function.

[0032] Step 133: generating the cross-level correlation loss function according to the spatial offset error and the semantic classification error, the cross-level correlation loss function containing error weighted accumulation results of multiple levels, and the weight of each level being dynamically adjusted by the weight coefficient of the corresponding level in the dynamic weight distribution matrix; wherein the spatial offset error and the semantic classification error are normalized by a preset balancing factor before weighted accumulation, so that the two types of errors have consistent dimensions.

[0033] Optionally, after calculating the spatial offset error and the semantic classification error of each level, a cross-level correlation loss function is generated, which performs weighted accumulation on the errors of multiple levels. The weight of each level is dynamically adjusted by the weight coefficient of the corresponding level in the dynamic weight distribution matrix. For example, for a level with high spatial resolution and important features, its weight coefficient is larger, and its contribution to the loss function is also larger. Before weighted accumulation, in order to make the spatial offset error and the semantic classification error have the same dimension so as to be reasonably accumulated, they are normalized by using a preset balancing factor. The determination of the balancing factor is based on the statistical analysis of the error data and the requirements of the model. After normalization, the errors of each level are accumulated according to the weight coefficients to generate a cross-level correlation loss function, which comprehensively reflects the prediction errors of the model at different levels and provides accurate guidance for the optimization of the model.

[0034] Step 140: iteratively optimizing the initial image recognition model according to the cross-level correlation loss function to generate a trained image recognition model; the trained image recognition model is used to perform end-to-end spatial positioning and semantic analysis operations on the input image to be recognized.

[0035] The step iteratively optimizes the initial image recognition model by using the cross-level correlation loss function to obtain a trained image recognition model capable of accurately performing spatial positioning and semantic parsing operations.

[0036] In an example embodiment, the iterative optimization of the initial image recognition model according to the cross-level correlation loss function to generate a trained image recognition model includes: step 141, parameter binding the multi-level convolutional feature extraction layers of the initial image recognition model with the dynamic weight distribution matrix in the cross-level correlation loss function, and synchronously feeding back the error weighted accumulation results of multiple levels output by the cross-level correlation loss function to the convolution kernel parameters of the corresponding level through the back propagation link.

[0037] Optionally, the multi-level convolutional feature extraction layers of the initial image recognition model are parameter bound with the dynamic weight distribution matrix in the cross-level correlation loss function, so that when the cross-level correlation loss function calculates the error weighted accumulation results of multiple levels, the error information can be accurately fed back to the convolution kernel parameters of the corresponding level in the initial image recognition model through the back propagation link. The back propagation link is a gradient descent-based algorithm that can adjust the parameters of the convolution kernel according to the size and direction of the error, so that the model can reduce the error when predicting next time. For example, when the error of a certain level is large, the back propagation link will adjust the parameters of the convolution kernel of that level accordingly, change the convolution operation mode of the convolution kernel and the image features, and thus affect the subsequent feature extraction process, in order to reduce the error of that level in the next iteration.

[0038] Step 142: based on the weight coefficients of each level in the dynamic weight distribution matrix, performing gradient scaling operation on the feature extraction channels of different levels in the initial image recognition model, so that the semantic classification error gradient of the first type of level resolution feature map is used to preferentially update the full connection classifier parameters, and the spatial offset error gradient of the second type of level resolution feature map is used to preferentially update the convolution locator parameters; wherein the first type of level is a deep feature map with a spatial resolution lower than a preset threshold, and the second type of level is a shallow feature map with a spatial resolution not lower than the preset threshold.

[0039] In this embodiment, gradient scaling operation is performed on the feature extraction channels of different levels in the initial image recognition model according to the weight coefficients of each level in the dynamic weight distribution matrix.

[0040] For the first type of level, i.e. the deep feature map with a spatial resolution lower than a preset threshold, the semantic classification error gradient is preferentially used to update the full connection classifier parameters, because the deep feature map usually contains more abstract and more semantically representative information, which plays a key role in semantic classification.

[0041] For the second type of hierarchy, i.e., shallow feature maps with spatial resolution not lower than a preset threshold, the spatial offset error gradient is prioritized for updating the convolutional locator parameters. Shallow feature maps often contain more spatial position information and are more important for spatial positioning of the target.

[0042] Through the above gradient scaling operation, the model can be more targeted to optimize feature processing at different levels, improving the performance of the model in spatial positioning and semantic analysis.

[0043] Step 143: In each iteration, the compensated fusion feature vector set is used as the cross-level supervision signal for the next round of forward propagation, and the spatial annotation information of the sample image set is input as the real supervision signal into the backpropagation link of the fully connected classifier and the convolutional locator parameters. The spatial consistency measure value of the feature correlation analysis module is used to dynamically adjust the weight coefficients of the dynamic weight distribution matrix, and the weight contribution of the level with a spatial consistency measure value lower than a preset threshold in the loss function is reduced.

[0044] In each iteration, the compensated fusion feature vector set is used as the cross-level supervision signal for the next round of forward propagation, which helps the model better utilize the previously processed feature information in subsequent forward propagation, improving the prediction accuracy. At the same time, the spatial annotation information of the sample image set is input as the real supervision signal into the backpropagation link of the fully connected classifier and the convolutional locator parameters. The spatial consistency measure value calculated by the feature correlation analysis module is used to dynamically adjust the weight coefficients of the dynamic weight distribution matrix. If the spatial consistency measure value of a certain level is lower than a preset threshold, it indicates that the features of this level have poor synergy with other levels, and the weight contribution of this level in the loss function is reduced to reduce its impact on the overall loss, so that the model pays more attention to levels with better spatial consistency, optimizing the performance of the model.

[0045] Step 144: According to the adjusted weight coefficients of the dynamic weight distribution matrix, the recalibration of the layer-by-layer weighted aggregation operation of the multi-level convolutional feature set is performed to match the channel dimension of the cross-level fusion feature vector set with the output layer dimension of the initial image recognition model, and the recalibrated cross-level fusion feature vector set is used as the input of the adaptive loss function optimization model.

[0046] The weight coefficients adjusted according to the dynamic weight distribution matrix recalibrate the layer-by-layer weighted aggregation operation of the multi-level convolution feature set, so as to ensure that the channel dimension of the cross-level fusion feature vector set matches the output layer dimension of the initial image recognition model. Dimension matching is crucial for the normal operation and accurate calculation of the model. During the recalibration process, the way of weighted aggregation is adjusted according to the weight coefficients, so that the generated cross-level fusion feature vector set can be correctly input into the adaptive loss function optimization model. After recalibration, the cross-level fusion feature vector set is used as the input of the adaptive loss function optimization model, providing suitable data for the next round of loss calculation and model optimization.

[0047] Step 145: Repeat steps 141-144 until the weighted cumulative result of the spatial offset error and the semantic classification error of all levels in the cross-level correlation loss function decreases by less than the preset convergence threshold for N consecutive iterations. Stop iteration and freeze the fully connected classifier parameters and convolution locator parameters of the initial image recognition model to generate the trained image recognition model.

[0048] It can be understood that steps 141 to 144 are repeatedly executed to continuously optimize the initial image recognition model. In each iteration, the weighted cumulative result of the spatial offset error and the semantic classification error of all levels in the cross-level correlation loss function changes. When the weighted cumulative result decreases by less than the preset convergence threshold for N consecutive iterations, it means that the optimization of the model has reached a stable state, and further iteration may not bring significant performance improvement. At this time, the iteration process is stopped, and the fully connected classifier parameters and convolution locator parameters of the initial image recognition model are frozen. These parameters, after multiple iterations of optimization, form a trained image recognition model, which has the ability to perform end-to-end spatial positioning and semantic analysis operations on the input image.

[0049] In an alternative embodiment, the method further comprises: step 146: linking the output end of the multi-level convolution feature set of the trained image recognition model with the execution end of the dynamic feature fusion mechanism; after the link is solidified, the weight coefficients of the dynamic weight distribution matrix are generated and fixed according to the statistical characteristics of the training data, and no longer change with the input data. The cross-level correlation loss function automatically inherits the dynamic weight distribution matrix and feature compensation parameters in the training phase in the inference phase to realize the output of the end-to-end spatial positioning and semantic analysis operation.

[0050] It can be understood that the output end of the multi-level convolution feature set of the trained image recognition model is linked and solidified with the execution end of the dynamic feature fusion mechanism. Link solidification makes the structure of the model more stable and improves the running efficiency. After completing the link solidification, the weight coefficients of the dynamic weight distribution matrix are determined and fixed according to the statistical characteristics of the training data. These weight coefficients are derived from the characteristics of a large amount of data and the performance of the model during the training process, and reflect the importance and correlation of different levels of features.

[0051] In the inference phase, the cross-level correlation loss function automatically inherits the dynamic weight distribution matrix and the feature compensation parameters in the training phase, which means that the model can utilize the knowledge and parameters learned in the training phase to quickly and accurately realize the output of end-to-end spatial positioning and semantic analysis operations when processing new input images, thereby improving the inference efficiency and accuracy of the model.

[0052] On the basis of the above, the method further includes an application step of the image recognition model: step 210: receiving an image to be identified, performing region division processing on the image to be identified according to the same image region division unit configuration as the sample image set, and generating a set of image region division units to be processed.

[0053] When the image to be identified is received, region division processing is performed thereon according to the same image region division unit configuration as the sample image set, in order to ensure that the subsequent feature extraction and processing process is consistent with the training phase. For example, the sample image is divided into a certain number and size of image region division units in the training phase, and the image to be identified is also divided in the same way. The division process divides the image to be identified into a plurality of image region division units to be processed according to pre-set rules and algorithms, and these units constitute a set of image region division units to be processed, providing basic data for subsequent feature extraction.

[0054] Step 220: processing the set of image region division units to be processed by the spatial perception feature extraction module of the trained image recognition model, to generate a set of multi-level convolution features corresponding to the image to be identified.

[0055] In this step, the spatial perception feature extraction module of the trained image recognition model is used to process the set of image region division units. The module uses the same operation mode as in the training phase, including using a pre-trained convolution feature extraction network, performing multi-level convolution kernel sliding operations, feature channel enhancement processing, and cross-modal feature alignment operations. Through these operations, different levels of feature information are extracted from each image region division unit to be processed, and a set of multi-level convolution features to be processed corresponding to the image to be recognized is generated. This set contains the feature expression of the image to be recognized at different perception granularities, providing rich data for subsequent analysis and recognition.

[0056] Step 230: input the set of multi-level convolution features to be processed into the dynamic feature fusion mechanism of the trained image recognition model, perform cross-level weighted aggregation operations based on the dynamic weight distribution matrix inherited from the training phase, and generate target spatial positioning results and initial semantic analysis results.

[0057] The set of multi-level convolution features to be processed is input into the dynamic feature fusion mechanism of the trained image recognition model. The mechanism performs cross-level weighted aggregation operations on the multi-level convolution features based on the dynamic weight distribution matrix inherited from the training phase. In this process, the features of different levels are weighted according to the weight coefficients corresponding to each level in the dynamic weight distribution matrix. For example, for a level with a larger weight coefficient, it is given a larger weight in the aggregation process, so that it occupies a more important position in the final result. Through the above weighted aggregation operation, the feature information of different levels is fused to generate target spatial positioning results and initial semantic analysis results. The target spatial positioning results clearly indicate the spatial position information of the target objects that may exist in the image to be recognized, and the initial semantic analysis results preliminarily judge the categories of these target objects.

[0058] Step 240: perform a spatial position matching degree verification operation on the initial semantic analysis results according to the target spatial positioning results, remove the semantic analysis entries in the initial semantic analysis results that do not overlap with the spatial coordinate range of the target spatial positioning results, and generate a verified semantic analysis result.

[0059] It can be understood that for each semantic parsing entry in the initial semantic parsing result, it is checked whether the corresponding target object matches the spatial coordinate range determined by the target spatial positioning result in space. For example, if it is indicated in the initial semantic parsing result that a certain object belongs to a certain category, but the spatial position corresponding to the semantic parsing of the object has no overlapping part with the spatial coordinate range of the object marked in the target spatial positioning result, then the semantic parsing entry is removed. Through the above strict spatial position matching degree checking operation, unreasonable semantic parsing information is removed, a more accurate and reliable checked semantic parsing result is generated, and the accuracy of semantic parsing is improved.

[0060] Step 250: performing combination and packaging processing on the checked semantic parsing result and the target spatial positioning result according to a preset corresponding relationship between a spatial label and a semantic label, to generate a target recognition result containing target detection frame coordinates and classification labels.

[0061] It can be understood that the checked semantic parsing result and the target spatial positioning result are integrated according to the preset corresponding relationship between the spatial label and the semantic label. The spatial label clearly indicates the spatial position representation of the target object in the image, and the semantic label corresponds to the category information of the target object. The combination and packaging of the two, for example, the association of the coordinate information of the target detection frame and the corresponding classification label, form a complete information unit. Finally, a target recognition result containing target detection frame coordinates and classification labels is generated, which clearly shows the spatial position and the category of each target object in the image to be recognized, and completes the recognition task of the input image.

[0062] In an optional embodiment, after generating the target recognition result containing the target detection frame coordinates and the classification labels, at least one of steps 310, 320 and 330 is further included: step 310: extracting the initial classification confidence of each target detection frame in the target recognition result, and screening out a first candidate detection frame set according to a preset confidence threshold; performing coordinate weighted average operation on the detection frames in the first candidate detection frame set whose spatial overlapping regions exceed a preset proportion, to generate a second candidate detection frame set after merging; inputting the second candidate detection frame set into a pre-trained detection frame regressor, to generate a scale-normalized optimized detection frame set based on the spatial annotation information in the sample image set; performing spatial position matching mapping on the optimized detection frame set and the classification labels in the target recognition result, to generate an updated target recognition result.

[0063] It can be understood that the initial classification confidence of each target detection box is extracted from the target recognition result, which reflects the accuracy of the model in classifying the target object corresponding to each detection box. According to the preset confidence threshold, those detection boxes with high confidence are selected to form the first candidate detection box set. For example, the confidence threshold is set to 0.8, and only the detection boxes with an initial classification confidence greater than 0.8 will be selected into the first candidate detection box set.

[0064] In the first candidate detection box set, the spatial overlap between each detection box is checked. For detection boxes with spatial overlap exceeding a preset proportion, in order to avoid repeated detection and improve the accuracy of detection, a coordinate weighted average operation is performed. For example, for two overlapping detection boxes, the weights are determined according to their respective areas, confidence and other factors, and the coordinates are weighted and averaged to generate a new coordinate position, and the second candidate detection box set is formed by merging. The second candidate detection box set is input into the pre-trained detection box regressor, which is trained based on the spatial annotation information in the sample image set and can normalize the scale of the detection box to generate an optimized detection box set.

[0065] Finally, the optimized detection box set is spatially and positionally matched with the classification labels in the target recognition result to ensure that each detection box corresponds to the correct classification label, and an updated target recognition result is generated to further improve the quality of target recognition.

[0066] Step 320: Based on the target detection box coordinates, extract the local image region corresponding to each classification label to generate a label-associated region feature vector; calculate the similarity measure value between the label-associated region feature vector and the preset semantic category prototype vector, and filter out abnormal classification labels with a similarity measure value below a dynamic threshold; perform cross-region semantic association analysis on the remaining classification labels, and merge classification labels that are spatially adjacent and have a semantic similarity satisfying a preset condition into a unified semantic label group; update the classification label description in the target recognition result according to the unified semantic label group to generate a semantic correction result.

[0067] Optionally, according to the coordinates of the target detection frame, a local image region corresponding to each classification label is extracted from the image to be identified, which contains feature information related to the classification label. By processing it, a label-associated region feature vector is generated, which can represent the features of the local image region. Then, the similarity measure value between the label-associated region feature vector and the preset semantic category prototype vector is calculated. The semantic category prototype vector is predefined and represents the typical feature vector of different semantic categories. By calculating the similarity, the matching degree of the features of the current label-associated region with each semantic category prototype can be determined. A dynamic threshold is set to filter out abnormal classification labels with a similarity measure value below the threshold. These abnormal classification labels may be inaccurate classification due to model misjudgment or image noise. For the remaining classification labels, cross-region semantic association analysis is performed. The spatially adjacent classification labels are checked to determine whether their semantic similarity meets the preset condition. If the condition is met, these classification labels are merged into a unified semantic label group. For example, if the semantic similarity of adjacent "car" and "sedan" classification labels reaches a certain degree, they can be merged into a "car" unified semantic label group. Finally, the classification label description in the target recognition result is updated according to the unified semantic label group to generate a semantic correction result, making the classification label more accurate and reasonable.

[0068] Step 330: Obtain a geographic information database associated with the image to be identified, retrieve matching geographic location metadata based on the target detection frame coordinates; logically verify the geographic location metadata and the classification label to generate a coordinate correction vector to adjust the boundary coordinates of the target detection frame; extract the timestamp information of the image to be identified, and combine the classification label sequence of the same geographic coordinate region in the historical recognition result to perform time continuity analysis; dynamically weight the confidence of the current classification label according to the time continuity analysis result to generate a spatiotemporal enhanced recognition result.

[0069] In this step, a geographic information database associated with the image to be recognized is obtained, which contains detailed information of various geographic locations. According to the coordinates of the target detection box, the geographic location metadata matching it is retrieved in the geographic information database. For example, the location corresponding to the target detection box may correspond to specific street names, area types, etc. in the database. These geographic location metadata are logically checked with the classification label. For example, if a certain target object is detected to be classified as a "beach chair", and its corresponding geographic location metadata shows that the location is in the high-rise building area in the city center, there may be an inaccurate classification. Through the above logical check, a coordinate correction vector is generated to adjust the boundary coordinates of the target detection box to make it more consistent with the actual situation. At the same time, the timestamp information of the image to be recognized is extracted, and the classification label sequence of the same geographic coordinate area in the historical recognition result is combined for time continuity analysis. For example, in a certain geographic area, a certain type of object often appears in a certain time period in the past, and the classification label of the object in the current recognition result does not match the historical situation, so the accuracy of the current classification needs to be considered. According to the result of time continuity analysis, the confidence of the current classification label is dynamically weighted. If the historical data shows that there is a high possibility that a certain object should appear in the area at the current time point, the confidence of the object classification label is appropriately increased; otherwise, it is decreased. Through the above method, the time and space factors are considered comprehensively to generate a spatiotemporal enhanced recognition result, improving the accuracy and reliability of image recognition.

[0070] In an independent embodiment, after generating the trained image recognition model, steps 150 or 160 are further included.

[0071] Step 150: adding a preset intensity of adversarial disturbance noise to each sample image in the sample image set to generate an adversarial training sample set; extracting an adversarial multi-level convolutional feature set of the adversarial training sample set through the trained image recognition model; calculating a feature difference measure value between the adversarial multi-level convolutional feature set and the corresponding original multi-level convolutional feature set to generate an adversarial robustness loss function; weighting and fusing the adversarial robustness loss function and the cross-level association loss function to generate a joint optimization loss function; unfreezing the fully connected classifier parameters and the convolutional locator parameters of the trained image recognition model, and performing optimization on the trained image recognition model according to the joint optimization loss function to generate an adversarially trained image recognition model; and after optimization, re-freezing the fully connected classifier parameters and the convolutional locator parameters.

[0072] In this embodiment, a preset intensity of adversarial noise is added to each sample image in the sample image set, and the addition of the noise is to simulate the interference factors that may occur in actual application and improve the robustness of the model. The preset intensity of noise is determined according to experiments and experience, which can cause certain interference to the image and does not cause complete loss of image information. After adding the noise, an adversarial training sample set is generated. The trained image recognition model is used to process the adversarial training sample set, and an adversarial multi-level convolutional feature set is extracted, which contains the feature information of the sample image after adding the noise. The feature difference measure value between the adversarial multi-level convolutional feature set and the corresponding original multi-level convolutional feature set is calculated. The difference between the two sets can be measured by using common algorithms and calculation methods, so as to generate an adversarial robustness loss function, which reflects the performance change of the model when facing adversarial perturbations. The adversarial robustness loss function and the cross-level correlation loss function are weighted and fused. According to the importance of the two loss functions and the requirements of the model, appropriate weights are determined, and they are added to generate a joint optimization loss function. At this time, the fully connected classifier parameters and the convolutional locator parameters of the trained image recognition model are unfrozen, so that these parameters can be adjusted according to the joint optimization loss function. The model is optimized by using the joint optimization loss function, and the parameters of the model are adjusted by using algorithms such as back propagation, so that the model can also maintain good performance in the case of adversarial perturbations. After optimization, the fully connected classifier parameters and the convolutional locator parameters are re-frozen, the structure and parameters of the model are fixed, and the adversarial trained image recognition model is generated, which improves the anti-interference ability and robustness of the model.

[0073] Step 160: generating an importance distribution map based on the importance score of the convolution kernel parameters corresponding to the multi-level convolutional feature set in the trained image recognition model; performing a pruning operation on the convolution kernel below the preset score threshold according to the importance distribution map to generate a sparse convolutional feature extraction network; performing a quantization encoding operation on the weight parameters of the sparse convolutional feature extraction network to generate a quantized image recognition model; unfreezing the fully connected classifier parameters and the convolutional locator parameters of the trained image recognition model, performing calibration fine-tuning processing on the quantized image recognition model based on the target image training data set to recover the accuracy loss caused by pruning and quantization, and generating a lightweight image recognition model; and re-freezing the fully connected classifier parameters and the convolutional locator parameters after calibration.

[0074] In this embodiment, first, the importance scores of the convolution kernel parameters corresponding to the multi-level convolution feature set in the trained image recognition model are scored. By analyzing the contribution of the convolution kernel to feature extraction, the influence on the performance of the model and other factors, an importance score value is assigned to each convolution kernel parameter. According to these score values, an importance distribution map is generated to intuitively show the importance of each convolution kernel parameter. According to the importance distribution map, a preset score threshold is set. For the convolution kernels below the threshold, pruning operation is performed, that is, these convolution kernels are removed from the model to generate a sparse convolution feature extraction network, which can reduce the number of parameters of the model and reduce the computational complexity of the model. The weight parameters of the sparse convolution feature extraction network are subjected to quantization encoding operation. By mapping the weight parameters to a limited set of discrete values, the representation accuracy of the weight parameters is reduced, thereby further compressing the size of the model to generate a quantized image recognition model. At this time, the fully connected classifier parameters and the convolution locator parameters of the trained image recognition model are unfrozen, and the quantized image recognition model is calibrated and fine-tuned using the target image training dataset. During the calibration process, the accuracy loss caused by pruning and quantization is recovered as much as possible by adjusting the parameters. After calibration, the fully connected classifier parameters and the convolution locator parameters are re-frozen, the structure and parameters of the model are fixed, and a lightweight image recognition model is generated. The model reduces the size and computational complexity of the model while maintaining a certain recognition accuracy, and is more suitable for running in resource-constrained environments.

[0075] To facilitate understanding of the above technical solutions, another application scenario is exemplarily introduced below.

[0076] In the field of intelligent security monitoring, it is necessary to accurately detect and recognize the targets in the images captured by the monitoring cameras to ensure the safety of the area.

[0077] In this security monitoring scenario, the target image training dataset is derived from a large number of images collected by multiple monitoring cameras at different time periods and under different weather conditions, which cover various possible scenarios in the monitoring area, such as personnel activities, vehicle access, abnormal behaviors, etc.

[0078] For each sample image, accurate spatial annotation is required. Common annotation tools can be used to annotate the targets in the image according to strict annotation specifications. For example, for a personnel target, the rectangular frame coordinate range of the personnel in the image is annotated, and the category of the personnel, such as "pedestrian" or "security guard", is also marked. For a vehicle target, the position coordinate range of the vehicle and the type of the vehicle, such as "sedan" or "truck", are annotated. In this way, the accuracy and consistency of the annotation information are ensured, and a high-quality target image training dataset is constructed.

[0079] Next, a spatial perception feature extraction operation is performed. Each sample image is divided into a preset number of image region division units. For example, according to the characteristics of the monitoring scene and the requirements of target detection, each image is divided into a 14x14 grid as an image region division unit. Each grid unit corresponds to a spatial detection region in the target image training data set, for example, some grid units may correspond to the area where personnel often appear, and others may correspond to the vehicle parking area, etc.

[0080] A pre-trained convolutional feature extraction network is called to perform a multi-level convolution kernel sliding operation on each image region division unit. The pre-trained convolutional feature extraction network is trained based on a large amount of security monitoring related image data and has image feature extraction capability. For each image region division unit, different size convolution kernels are used for multi-level sliding. For example, a 3x3 convolution kernel is used to capture detailed features, and a 5x5 convolution kernel is used to focus on more macro structural features. During the sliding process, the convolution kernel and the pixels of the image region division unit perform multiplication and accumulation operations at corresponding positions to generate an initial convolution feature map.

[0081] Feature channel enhancement processing is performed on the initial convolution feature map. The initial convolution feature map is usually composed of multiple feature channels. According to the statistical characteristics of image data and model requirements, a preset channel attention weight distribution table is determined. For example, a certain feature channel is mainly responsible for capturing the contour features of personnel, and according to experience and data statistics, a higher weight is assigned to this channel. For some channels containing noise or less important information, a lower weight is assigned. According to this weight distribution table, each feature channel is weighted, that is, each channel element of the initial convolution feature map is multiplied by the corresponding weight value. Through this dynamic weighted fusion method, an enhanced convolution feature map is generated.

[0082] The enhanced convolution feature map and the corresponding image region division unit are subjected to cross-modal feature alignment operation. By calculating the similarity index between the two, such as cosine similarity, the features are adjusted and aligned, so that the feature information in the convolution feature map and the spatial position information of the image region division unit are better matched and fused, and finally a multi-level convolution feature set with spatial perception features is generated.

[0083] The multi-level convolutional feature set is input into a preset adaptive loss function optimization model, and a cross-level correlation loss function is generated by combining a dynamic feature fusion mechanism. Based on the spatial resolution corresponding to the feature expression of each level in the multi-level convolutional feature set, a dynamic weight distribution matrix matched with the level granularity is generated. For example, in the security monitoring model, the levels with higher spatial resolution contain more information about the target details, and are more critical to the accurate identification of the target, so these levels are given relatively higher weights. Through the dynamic weight distribution matrix, the feature expressions of different levels in the multi-level convolutional feature set are weighted and aggregated layer by layer to generate a cross-level fusion feature vector set.

[0084] The cross-level fusion feature vector set is input into a feature correlation analysis module in the adaptive loss function optimization model to extract spatial consistency measurement values between different level feature vectors. For example, the spatial consistency is measured by calculating the Euclidean distance between the feature vectors. According to these measurement values, a difference compensation correction operation is performed on the cross-level fusion feature vector set. If the spatial consistency of the feature vector of a certain level with other levels is poor, it is adjusted accordingly to improve the overall spatial consistency. After generating the compensated fusion feature vector set, it is compared with the original feature expressions in the multi-level convolutional feature set layer by layer, and the spatial offset error and semantic classification error of each level feature expression are calculated.

[0085] A cross-level correlation loss function is generated according to the spatial offset error and the semantic classification error. In the security monitoring scenario, it is very important to accurately identify the target and locate the target position, so appropriate weights are set for the spatial offset error and the semantic classification error when generating the loss function. For example, after multiple experiments and analysis, it is determined that the weight of the spatial offset error is set to 4 and the weight of the semantic classification error is set to 3. At the same time, in order to make the dimensions of the two types of errors consistent, a preset balancing factor is used to normalize them. For example, by calculating the standard deviation of the error data, the balancing factor is determined, and the errors of each level are accumulated according to the weight coefficients to generate the cross-level correlation loss function.

[0086] The initial image recognition model is iteratively optimized according to the cross-level correlation loss function. The multi-level convolutional feature extraction layer of the initial image recognition model is parameter-bound with the dynamic weight distribution matrix in the cross-level correlation loss function, and the error weighted accumulation results of multiple levels output by the cross-level correlation loss function are synchronously fed back to the convolution kernel parameters of the corresponding levels through the back propagation link. For example, when the spatial offset error of a certain level is large, the back propagation link adjusts the parameters of the level convolution kernel to change the convolution operation mode of the convolution kernel and the image features, so as to reduce the error of the level in the next iteration.

[0087] Based on the weight coefficients of each level in the dynamic weight distribution matrix, gradient scaling operations are performed on the feature extraction channels of different levels in the initial image recognition model. For deep feature maps with a spatial resolution lower than a preset threshold, their semantic classification error gradients are prioritized for updating the parameters of the fully connected classifier, because deep feature maps usually contain more abstract and semantically representative information, which plays a key role in accurate classification. For shallow feature maps with a spatial resolution not lower than the preset threshold, their spatial offset error gradients are prioritized for updating the parameters of the convolutional locator, because shallow feature maps tend to contain more spatial position information, which is more important for the spatial positioning of the target.

[0088] In each iteration process, the compensated fusion feature vector set is used as the cross-level supervision signal for the next round of forward propagation, and the spatial annotation information of the sample image set is used as the real supervision signal to input into the backpropagation link of the fully connected classifier and the convolutional locator parameters. The weight coefficients of the dynamic weight distribution matrix are dynamically adjusted by the spatial consistency measurement value of the feature correlation analysis module. If the spatial consistency measurement value of a certain level is lower than a preset threshold, it indicates that the features of this level have poor cooperativeness with other levels. At this time, the weight contribution of this level in the loss function is reduced, and its influence on the overall loss is reduced, so that the model pays more attention to the levels with better spatial consistency, and the performance of the model is optimized.

[0089] According to the adjusted weight coefficients of the dynamic weight distribution matrix, the recalibration of the weighted aggregation operation of the multi-level convolutional feature set is performed, so that the channel dimension of the cross-level fusion feature vector set matches the output layer dimension of the initial image recognition model, and the recalibrated cross-level fusion feature vector set is used as the input of the adaptive loss function optimization model. Repeat the above iteration optimization steps until the weighted cumulative result of the spatial offset error and the semantic classification error of all levels in the cross-level correlation loss function decreases by less than a preset convergence threshold (for example, the preset convergence threshold is 0.01) for 5 consecutive iterations. Stop iteration and freeze the fully connected classifier parameters and convolutional locator parameters of the initial image recognition model, and generate a trained image recognition model.

[0090] After training the completed image recognition model, the application stage of the model is entered. When receiving a monitoring image to be recognized, the to-be-recognized image is divided into 14x14 grids according to the same image region division unit configuration as the sample image set, generating a set of to-be-processed image region division units.

[0091] The spatial perception feature extraction module of the trained image recognition model processes the to-be-processed image region division unit set. The module uses the same operation mode as in the training stage, including using a pre-trained convolution feature extraction network, performing multi-level convolution kernel sliding operations, feature channel enhancement processing, and cross-modal feature alignment operations, etc., to generate a to-be-processed multi-level convolution feature set corresponding to the to-be-recognized image.

[0092] The to-be-processed multi-level convolution feature set is input into the dynamic feature fusion mechanism of the trained image recognition model, and cross-level weighted aggregation operations are performed based on the dynamic weight distribution matrix inherited from the training stage. For example, according to the weight coefficients corresponding to each level in the dynamic weight distribution matrix, the features of different levels are weighted. For level features with larger weight coefficients, they are given greater weight in the aggregation process, so that they occupy a more important position in the final result. Through the above weighted aggregation operation, the target spatial positioning result and the initial semantic analysis result are generated. The target spatial positioning result clearly shows the spatial position information of the target objects that may exist in the to-be-recognized image, such as the specific coordinate range of a person or a vehicle in the image; the initial semantic analysis result preliminarily judges the categories of these target objects, such as "pedestrian" or "car".

[0093] According to the target spatial positioning result, a spatial position matching degree verification operation is performed on the initial semantic analysis result. For each semantic analysis entry in the initial semantic analysis result, it is checked whether the corresponding target object in space matches the spatial coordinate range determined by the target spatial positioning result. For example, if the initial semantic analysis result indicates that a certain object belongs to the "pedestrian" category, but the spatial position corresponding to the semantic analysis of the object does not have an overlapping part with the spatial coordinate range of the object marked in the target spatial positioning result, then the semantic analysis entry is removed, and the verified semantic analysis result is generated.

[0094] The verified semantic analysis result and the target spatial positioning result are combined and packaged according to the preset correspondence between the spatial label and the semantic label. For example, the coordinate information of the target detection box is associated with the corresponding classification label to form a complete information unit, and a target recognition result containing the target detection box coordinates and the classification label is generated, which clearly shows the spatial position and the category of each target object in the to-be-recognized image, and completes the recognition task of the input monitoring image.

[0095] After generating the target recognition result containing the target bounding box coordinates and classification labels, further optimization processing can be performed. For example, the initial classification confidence of each target bounding box in the target recognition result is extracted, and a preset confidence threshold of 0.7 is set. According to this threshold, a first candidate bounding box set is selected, that is, only the bounding boxes with an initial classification confidence greater than 0.7 are selected. In the first candidate bounding box set, the spatial overlap between the bounding boxes is checked. For bounding boxes with a spatial overlap exceeding a preset proportion (for example, the preset proportion is 0.5), a coordinate weighted average operation is performed. For example, for two overlapping bounding boxes, the weights are determined according to their respective areas, confidence, and other factors, and the coordinates are weighted and averaged to generate a new coordinate position, and the second candidate bounding box set is formed. The second candidate bounding box set is input into a pre-trained bounding box regressor, which is trained based on the spatial annotation information in the sample image set and can normalize the scale of the bounding box to generate an optimized bounding box set. Finally, the optimized bounding box set is spatially and positionally matched with the classification labels in the target recognition result to ensure that each bounding box corresponds to the correct classification label, and an updated target recognition result is generated to further improve the quality of target recognition.

[0096] In addition, based on the target bounding box coordinates, the local image region corresponding to each classification label is extracted to generate a label-associated region feature vector. The similarity measure value between the label-associated region feature vector and the preset semantic category prototype vector is calculated, for example, a dynamic threshold of 0.6 is set, and abnormal classification labels with a similarity measure value below the threshold are filtered out. For the remaining classification labels, cross-region semantic association analysis is performed. The spatially adjacent classification labels are checked to determine whether their semantic similarity meets the preset conditions. If the conditions are met, these classification labels are merged into a unified semantic label group. For example, for adjacent "male pedestrian" and "female pedestrian" classification labels, if their semantic similarity reaches a certain degree, they can be merged into a "pedestrian" unified semantic label group. Finally, the classification label description in the target recognition result is updated according to the unified semantic label group to generate a semantic correction result, making the classification labels more accurate and reasonable.

[0097] Meanwhile, a geographic information database associated with the image to be recognized is acquired, and matching geographic location metadata is retrieved based on the target detection frame coordinates. For example, the location corresponding to the target detection frame can correspond to specific street names, area types, and other information in the database. These geographic location metadata are logically checked with the classification label. For example, if a certain target object is detected to be classified as "suspicious person", and its corresponding geographic location metadata shows that the location is in a normal office area, there may be an inaccurate classification. Through the above logical check, a coordinate correction vector is generated to adjust the boundary coordinates of the target detection frame to make it more consistent with the actual situation. The timestamp information of the image to be recognized is extracted, and the classification label sequence of the same geographic coordinate area in the historical recognition result is combined for time continuity analysis. For example, in a certain geographic area, a large number of "pedestrians" and "vehicles" often appear during working hours in the past, but the current recognition result shows that "suspicious person" appears in the same area at the same time point, so the accuracy of the current classification needs to be considered. According to the result of the time continuity analysis, the confidence of the current classification label is dynamically weighted. If the historical data shows that the area should have a higher probability of appearing a certain object at the current time point, the confidence of the object classification label is appropriately increased; otherwise, it is decreased. Through the above method, time and space factors are considered to generate a spatiotemporal enhanced recognition result, improving the accuracy and reliability of image recognition.

[0098] In addition, after generating the trained image recognition model, an adversarial training or model lightweight processing can be performed. For example, an adversarial disturbance noise of a preset intensity is added to each sample image in the sample image set to generate an adversarial training sample set. The noise of the preset intensity is determined according to experiments and experience, which can cause certain interference to the image without completely losing the image information. The adversarial multi-level convolution feature set of the adversarial training sample set is extracted by the trained image recognition model, the feature difference measure value between the adversarial multi-level convolution feature set and the corresponding original multi-level convolution feature set is calculated, and an adversarial robustness loss function is generated. The adversarial robustness loss function and the cross-level correlation loss function are weighted and fused, for example, the weight of the adversarial robustness loss function is 0.3, and the weight of the cross-level correlation loss function is 0.7, to generate a joint optimization loss function. The fully connected classifier parameters and the convolution locator parameters of the trained image recognition model are unfrozen, the model is optimized according to the joint optimization loss function, and an adversarial trained image recognition model is generated, improving the anti-interference ability and robustness of the model.

[0099] Or based on the importance score of the convolution kernel parameter corresponding to the multi-level convolution feature set in the trained image recognition model, an importance distribution atlas is generated. According to the importance distribution atlas, a pruning operation is performed on the convolution kernel below the preset score threshold (for example, the preset score threshold is 0.3), and a sparse convolution feature extraction network is generated. The weight parameters of the sparse convolution feature extraction network are subjected to a quantization encoding operation, and a quantized image recognition model is generated. The frozen fully connected classifier parameters and convolution locator parameters of the trained image recognition model are thawed, and the quantized image recognition model is subjected to calibration fine-tuning processing based on the target image training data set, to recover the precision loss caused by pruning and quantization, and generate a lightweight image recognition model, so that the model reduces the size and calculation amount of the model under the premise of maintaining certain recognition accuracy, and is more suitable for running on resource-limited monitoring devices.

[0100] It is worth mentioning that when implementing the above technical solutions, the skilled person in the art can perform feature extraction and integration on the image division region based on the region of interest feature alignment method (RoI Align) in the prior art, generate a fixed-size region feature expression through bilinear interpolation to retain spatial context information, and avoid semantic fragmentation caused by region division.

[0101] For the feature alignment process, the alignment logic can be reconstructed in combination with the cross-level spatial attention mechanism to achieve adaptive fusion by calculating the spatial correlation weight between different resolution feature maps.

[0102] To solve the redundancy problem of end-to-end model prediction and post-processing verification, a non-maximum suppression (NMS) and confidence threshold screening joint strategy can be introduced to dynamically filter low-quality prediction results based on the intersection over union (IoU) of the target detection box and the classification confidence, and maintain the prediction consistency of the end-to-end architecture; for the semantic prototype vector missing problem, the prototype network technology can be used to dynamically generate class prototypes in the training stage through clustering algorithm, and the feature vector and prototype distance are included in the loss function, to ensure that the similarity measurement in the application stage has training basis.

[0103] For the logical fault of geographic information verification and training data, a multi-task learning framework can be used to embed geographic metadata into model training, map the geographic location to a low-dimensional vector through a spatial coordinate encoding layer and fuse it with the image features, so that the model can autonomously mine the implicit association between geographic attributes and target categories.

[0104] In terms of adversarial training process optimization, an adversarial sample can be generated by projection gradient descent (PGD) and trained synchronously with the original sample, and the robustness of the model to noise interference is enhanced by using feature similarity constraints; in terms of feature map dimension difference, the resolution and distribution range of features at different levels can be unified by combining the multi-scale fusion mechanism of feature pyramid network (FPN) and the grouping normalization technology, and the spatial offset error and semantic classification error are dynamically weighted and balanced by using gradient normalization algorithm (Grad Norm), the standardization coefficient is automatically adjusted by monitoring the loss gradient in real time, and stable optimization of cross-level error and improvement of overall performance of the model are realized.

[0105] In addition, for the selection of the loss function mentioned above, those skilled in the art can flexibly select specific and adaptive loss functions and combinations thereof according to actual needs to clearly and completely implement the embodiments of the present application.

[0106] In summary, please refer to Figure 2 The embodiments of the present application innovatively obtain a sample image set with spatial annotation information as a training data set, and effectively solve the problem of insufficient accuracy and comprehensiveness of spatial positioning and semantic analysis in image recognition by feature extraction, adaptive loss function optimization and dynamic feature fusion mechanism, thereby improving the performance of the image recognition model (for example, accurately identifying different categories of targets "dog", "bicycle" and "car" in Figure 2

[0107] The embodiments of the present application can provide rich and accurate data basis for subsequent processing by obtaining a target image training data set containing a sample image set with spatial annotation information; spatial perception feature extraction is performed on the sample image set, and the generated multi-level convolution feature set covers feature expressions of different perception granularities, which is helpful for capturing various feature details of the image; the feature set is input into an adaptive loss function optimization model, and a cross-level correlation loss function is generated by combining a dynamic feature fusion mechanism, which can effectively mine the correlation between features at different levels and provide more targeted guidance for model optimization; based on this, the initial image recognition model is iteratively optimized, so that the trained image recognition model can efficiently perform end-to-end spatial positioning and semantic analysis on the input image to be recognized, thereby significantly improving the accuracy and comprehensiveness of image recognition.

[0108] Referring to Figure 3 As shown in the figure, the figure is a schematic diagram of the basic structure of an image recognition model training device 200 provided by the embodiments of the present application, the image recognition model training device 200 comprises a processor 201, a storage device 202, wherein the storage device 202 stores a computer program 2020, and when the computer program 2020 is executed by the processor 201, the processor 201 implements any of the image recognition model training methods. ​

[0109] On the basis of the above, a readable storage medium is provided, and the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the steps of the above method.

[0110] In the technical solutions related to the above embodiments of the application, whether it is a multi-dimensional feature comparison calculation or a composite parameter construction, if there are problems caused by significant differences in the number of dimensions, units of dimensions, and semantic meanings of different features, those skilled in the art can fully understand that these differences need to be properly handled based on their professional knowledge and past practical experience, so that the calculation result is accurate and has comparability, and logical confusion, unclear mathematical meaning, and the like are avoided.

[0111] In detail, when facing features with different numbers of dimensions, in order to accurately calculate the similarity, matching degree, or feature distance between different features, those skilled in the art can use various strategies.

[0112] Feature selection is a commonly used method. For a high-dimensional feature set, a feature subset that is most representative and matches the number of low-dimensional features can be selected from the high-dimensional features according to indicators such as importance and correlation of the features. By using methods such as chi-square test and information gain for feature selection, the most valuable features for the technical solution are screened out, so that the high-dimensional features are reduced to a dimension comparable to the low-dimensional features, and then similarity or distance calculation is performed.

[0113] Feature extraction is also an effective means. By constructing a suitable feature extraction model, features of different dimensions are mapped to a common low-dimensional feature space. Principal component analysis (PCA) can not only be used to process dimensional differences, but also project high-dimensional features to a low-dimensional space composed of principal components, so that features of different dimensions have comparability in this low-dimensional space. In addition, deep learning models such as autoencoders can also be used for feature extraction, which can automatically learn the latent representation of input features and convert features of different dimensions into feature vectors of the same dimension for subsequent similarity, matching degree, or feature distance calculation.

[0114] In addition, a kernel method can also be used. The kernel function can calculate the similarity between features in a high-dimensional space without explicitly mapping the features to a high-dimensional space. For features with different numbers of dimensions, a suitable kernel function such as a Gaussian kernel function or a polynomial kernel function can be selected to directly calculate the similarity between them. This method avoids the difficulty of direct calculation caused by different feature dimensions and can effectively measure the relationship between features in the original feature space or an implicit high-dimensional space.

[0115] When processing the comparison of multi-dimensional features, in order to realize the comparability alignment of the feature space, those skilled in the art can use various existing general technical means.

[0116] Standardization is a widely used and effective method that transforms the original feature data into a standard normal distribution with a mean of 0 and a standard deviation of 1 through a specific linear transformation. This processing method can essentially eliminate the influence of the dimensions of different features, allowing all features to be compared on the same scale. For example, in a dataset containing features with different dimensions, after standardization preprocessing, these features can be calculated for similarity or distance on the same scale, avoiding calculation bias caused by different dimensions.

[0117] Mapping transformation is also an effective way to solve the problem of dimension difference. It can map the original features to a new space according to the specific properties of the features and actual business needs. In this new space, features with different dimensions can have better comparability. For features with nonlinear relationships, domain experts can use logarithmic transformation, power transformation, etc. to convert them to linear relationships, making it easier to calculate similarity or distance. For example, when dealing with features with exponential growth trends, logarithmic transformation can convert them to linear relationships, making subsequent calculations more accurate and convenient.

[0118] Space projection is also an important technical means that projects high-dimensional feature space into low-dimensional space while preserving important information between features as much as possible. By carefully selecting the projection direction and projection dimension, domain experts can reduce the influence of dimension difference on the calculation results while reducing the data dimension. Common space projection methods include Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), etc. Taking Principal Component Analysis as an example, it projects high-dimensional data into a low-dimensional space composed of principal components, simplifying the data structure while reducing the interference of dimension difference on feature comparison.

[0119] In the construction process of composite parameters (such as loss function values), different parameter items often have different dimensions. Domain experts can use normalization processing or adaptive weight distribution mechanism based on distribution characteristics.

[0120] Normalization processing is to unify the value range of different parameter items to a fixed interval, such as [0, 1]. This processing method can eliminate the influence of dimension difference and ensure that each parameter item has the same importance when weighted fusion. Common normalization methods include min-max normalization, Z-score normalization, etc. Taking min-max normalization as an example, it performs linear transformation on the parameter items to scale their value range to the [0, 1] interval, so that parameter items with different dimensions can be weighted and fused on the same standard.

[0121] The adaptive weight distribution mechanism based on distribution characteristics dynamically adjusts the weights of different parameter items according to their distribution characteristics. For parameter items with larger variances, the weight of the parameter item can be appropriately reduced by those skilled in the art; for parameter items with smaller variances, the weight of the parameter item can be appropriately increased by those skilled in the art. In this way, the composite loss function can pay more attention to the parameter items with smaller variances, thereby improving the stability and generalization ability of the model. For example, in a composite loss function containing multiple parameter items, if the variance of a certain parameter item is large, it means that the fluctuation of the parameter item is large, which may have an adverse effect on the stability of the model. At this time, reducing the weight of the parameter item can reduce the adverse effect; for parameter items with smaller variances, increasing the weight of the parameter item can make the model pay more attention to the information reflected by the parameter item, thereby improving the overall performance of the model.

[0122] The above-mentioned general technical means for solving the problems of feature matching and loss balancing belong to the common knowledge in the art. These technical means have been fully verified and widely used in a large number of practical applications. When facing similar dimensional difference problems, those skilled in the art can skillfully and flexibly use these methods to handle them.

[0123] It should be noted that the embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system or device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

Claims

1. A method for training an image recognition model, the method comprising: The method comprises the following steps: acquiring a target image training data set containing a sample image set with spatial annotation information; performing a spatial perception feature extraction operation on the sample image set to generate a multi-level convolution feature set of each sample image, the multi-level convolution feature set containing feature expressions of different perception granularities; inputting the multi-level convolution feature set into a preset adaptive loss function optimization model and combining a dynamic feature fusion mechanism to generate a cross-level correlation loss function; iteratively optimizing an initial image recognition model according to the cross-level correlation loss function to generate a trained image recognition model; and the trained image recognition model is used to perform end-to-end spatial positioning and semantic analysis operations on an input image to be recognized.

2. The method of claim 1, wherein, The method of performing a spatial perception feature extraction operation on the sample image set to generate a multi-level convolution feature set of each sample image comprises the following steps: dividing each sample image into a preset number of image region division units, each image region division unit corresponding to a spatial detection region in the target image training data set; calling a pre-trained convolution feature extraction network to perform a multi-level convolution kernel sliding operation on each image region division unit to generate an initial convolution feature map; performing feature channel enhancement processing on the initial convolution feature map, dynamically weighting and fusing the feature channels according to a preset channel attention weight distribution table to generate an enhanced convolution feature map; and performing a cross-modal feature alignment operation on the enhanced convolution feature map and the corresponding image region division unit to generate a multi-level convolution feature set with spatial perception features.

3. The method of claim 1, wherein, The multi-level convolutional feature set is input into a preset adaptive loss function optimization model, and a cross-level correlation loss function is generated by combining a dynamic feature fusion mechanism, including: based on the spatial resolution corresponding to the feature expression of each level in the multi-level convolutional feature set, a dynamic weight distribution matrix matched with the level granularity is generated; by using the dynamic weight distribution matrix, a layer-by-layer weighted aggregation operation is performed on the feature expressions of different levels in the multi-level convolutional feature set, to generate a cross-level fusion feature vector set; the cross-level fusion feature vector set is input into a feature correlation analysis module in the adaptive loss function optimization model, to extract the spatial consistency measure value between different level feature vectors; according to the spatial consistency measure value, a difference compensation correction operation is performed on the cross-level fusion feature vector set, to generate a compensated fusion feature vector set; the compensated fusion feature vector set is compared with the original feature expressions in the multi-level convolutional feature set layer by layer, to calculate the spatial offset error and semantic classification error of each level feature expression; according to the spatial offset error and the semantic classification error, the cross-level correlation loss function is generated, the cross-level correlation loss function contains the error weighted accumulation results of multiple levels, and the weight of each level is dynamically adjusted by the weight coefficient of the corresponding level in the dynamic weight distribution matrix; wherein the spatial offset error and the semantic classification error are normalized by a preset balancing factor before weighted accumulation, so that the dimensions of the two types of errors are consistent.

4. The method of claim 3, wherein, The iterative optimization processing of the initial image recognition model according to the cross-level correlation loss function to generate a trained image recognition model comprises: parameter binding of the multi-level convolution feature extraction layer of the initial image recognition model and the dynamic weight distribution matrix in the cross-level correlation loss function, error weighted accumulation results of multiple levels output by the cross-level correlation loss function are synchronously fed back to the convolution kernel parameters of the corresponding level through the back propagation link; based on the weight coefficients of each level in the dynamic weight distribution matrix, gradient scaling operation is performed on the feature extraction channels of different levels in the initial image recognition model, so that the semantic classification error gradient of the first type of level resolution feature map preferentially updates the full connection classifier parameter, and the spatial offset error gradient of the second type of level resolution feature map preferentially updates the convolution locator parameter; wherein the first type of level is a deep feature map with a spatial resolution lower than a preset threshold, and the second type of level is a shallow feature map with a spatial resolution not lower than the preset threshold; in each iteration process, the compensated fusion feature vector set is taken as the cross-level supervision signal of the next round of forward propagation, and the spatial annotation information of the sample image set is taken as the real supervision signal and input into the back propagation link of the full connection classifier and the convolution locator parameter, the weight coefficients of the dynamic weight distribution matrix are dynamically adjusted by the spatial consistency measurement value of the feature correlation analysis module, and the weight contribution of the level with a spatial consistency measurement value lower than a preset threshold in the loss function is reduced; according to the adjusted weight coefficients of the dynamic weight distribution matrix, the layer-by-layer weighted aggregation operation of the multi-level convolution feature set is recalibrated, so that the channel dimension of the cross-level fusion feature vector set matches the output layer dimension of the initial image recognition model, and the recalibrated cross-level fusion feature vector set is taken as the input of the adaptive loss function optimization model; the steps of parameter binding of the multi-level convolution feature extraction layer of the initial image recognition model and the dynamic weight distribution matrix in the cross-level correlation loss function to the step of taking the recalibrated cross-level fusion feature vector set as the input of the adaptive loss function optimization model are repeatedly performed until the weighted accumulation results of the spatial offset error and the semantic classification error of all levels in the cross-level correlation loss function are continuously reduced by less than a preset convergence threshold for N times of iteration, the iteration is stopped, and the full connection classifier parameter and the convolution locator parameter of the initial image recognition model are frozen to generate the trained image recognition model.

5. The method of claim 4, wherein, The method further comprises: link solidification of the output end of the multi-level convolution feature set of the trained image recognition model and the execution end of the dynamic feature fusion mechanism; after the link solidification is completed, the weight coefficients of the dynamic weight distribution matrix are generated and fixed according to the statistical characteristics of the training data, and no longer change with the input data, and the cross-level correlation loss function automatically inherits the dynamic weight distribution matrix and the feature compensation parameter in the training phase in the inference phase to realize the output of the end-to-end spatial positioning and semantic analysis operation.

6. The method of claim 1, wherein, The method further comprises: receiving an image to be identified, performing region division processing on the image to be identified according to the same image region division unit configuration as the sample image set, and generating a set of image region division units to be processed; processing the set of image region division units to be processed by the spatial perception feature extraction module of the trained image recognition model to generate a set of multi-level convolution features to be processed corresponding to the image to be identified; inputting the set of multi-level convolution features to be processed into the dynamic feature fusion mechanism of the trained image recognition model, performing cross-level weighted aggregation operation based on the dynamic weight distribution matrix inherited in the training stage to generate a target spatial positioning result and an initial semantic analysis result; performing spatial position matching degree checking operation on the initial semantic analysis result according to the target spatial positioning result, removing semantic analysis entries in the initial semantic analysis result that do not overlap with the spatial coordinate range of the target spatial positioning result, and generating a checked semantic analysis result; performing combination and packaging processing on the checked semantic analysis result and the target spatial positioning result according to the corresponding relationship between the preset spatial label and the semantic label to generate a target recognition result containing target detection frame coordinates and classification labels.

7. The method of claim 6, wherein, After generating the target recognition result containing the target detection frame coordinates and the classification labels, the method further comprises: extracting the initial classification confidence of each target detection frame in the target recognition result, and screening out a first candidate detection frame set according to a preset confidence threshold; performing coordinate weighted average operation on the detection frames in the first candidate detection frame set whose spatial overlapping regions exceed a preset proportion to generate a second candidate detection frame set after merging; inputting the second candidate detection frame set into a pre-trained detection frame regressor to generate a scale-normalized optimized detection frame set based on the spatial annotation information in the sample image set; performing spatial position matching mapping between the optimized detection frame set and the classification labels in the target recognition result to generate an updated target recognition result.

8. The method of claim 6, wherein, After generating the target recognition result containing the target detection frame coordinates and the classification labels, the method further comprises: extracting a local image region corresponding to each classification label based on the target detection frame coordinates to generate a label-associated region feature vector; calculating the similarity measure value between the label-associated region feature vector and a preset semantic category prototype vector, and filtering abnormal classification labels whose similarity measure values are lower than a dynamic threshold; performing cross-region semantic association analysis on the remaining classification labels, and merging classification labels that are spatially adjacent and have semantic similarity satisfying a preset condition into a unified semantic label group; updating the classification label description in the target recognition result according to the unified semantic label group to generate a semantic correction result.

9. The method of claim 6, wherein, After generating the target recognition result containing the target detection frame coordinates and the classification label, further comprising: acquiring a geographic information database associated with the image to be recognized, and retrieving matching geographic position metadata based on the target detection frame coordinates; logically verifying the geographic position metadata and the classification label to generate a coordinate correction vector to adjust the boundary coordinates of the target detection frame; extracting the timestamp information of the image to be recognized, and combining the classification label sequence of the same geographic coordinate area in the historical recognition result to perform time continuity analysis; dynamically weighting the confidence of the current classification label according to the time continuity analysis result to generate a spatiotemporal enhanced recognition result.

10. An apparatus for training an image recognition model, comprising: Comprise: a processor; a storage device having a computer program stored thereon, wherein when the computer program is executed by the processor, the processor implements the training method of the image recognition model according to any one of claims 1-9.

Images