A neural network-based image classification method and device

Abstract

The application discloses a kind of neural network-based image classification method and equipment, it is related to computer vision and image processing technical field, including obtaining scene image, by convolutional neural network according to the response difference of character candidate area and background area carries out cross-scale self-adapting weighted fusion, obtains the initial feature map containing spatial distribution information, according to the initial feature map containing spatial distribution information, using region proposal network carries out character region positioning and boundary frame regression processing, obtains the local receptive feature containing character boundary frame;The neural network-based image classification method and equipment can retain the features with higher classification contribution and suppress redundant information, thereby improving the accuracy, stability and discriminant ability of complex scene image classification containing characters.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to an image classification method and device based on neural networks. Background Technology

[0002] Image classification technology is an important research direction in the field of computer vision, and it is widely used in scenarios such as intelligent surveillance, social media content management, personalized recommendation, and public safety. As image classification tasks have gradually evolved from simple object recognition to complex social scene understanding, classification models not only need to identify environmental information in images, but also need to combine the identity features of people in images to make a comprehensive judgment on the semantic relationship between people and their surroundings.

[0003] Existing image classification methods typically use the entire image as a uniform input, extracting image features through neural networks to complete the classification. However, in complex scene images containing people, there are significant differences between the person region and the background region in terms of scale, spatial distribution, and feature response intensity. The person region usually occupies a small area of ​​the image, while the background region contains more environmental information. When directly extracting features from the entire image, the feature responses generated by the background region can easily mask the local features of the person region, resulting in insufficient representation of the person's identity information. Furthermore, features at different scales have varying representational capabilities for the person region and the background region. Shallower features can preserve local details such as facial texture and contours, but have limited ability to express the overall semantics of the scene. Deeper features can reflect the scene layout and contextual relationships, but are prone to losing fine-grained information about the person's region. Existing methods often lack effective adjustment for the response differences between the person's region and the background region when fusing features of different scales, which can easily cause information imbalance between features of different scales, affecting the accurate localization and feature extraction of the person's region. During the feature extraction process of the person's region, changes in the person's posture, shooting angle, person's size, and occlusion can also lead to inconsistent feature distribution of the same person in different images. If the person's region is not accurately located, or the boundary between the person's region and the background region is not clearly defined, some background information can be introduced into the person's features, or effective information of the person's edge region can be missed, thereby reducing the stability and discrimination ability of the person's identity features.

[0004] Furthermore, character identity features and scene environment features belong to different semantic levels. Character identity features mainly reflect local details of a person, while scene environment features mainly reflect the overall layout and contextual information of the image. Existing methods typically directly concatenate or superimpose these two types of features without fully considering the differences in spatial location, channel response, and discriminative contribution between the two types of features. This can easily lead to competition and mutual interference between character identity information and scene environment information during the fusion process. Moreover, the fused features often contain a lot of repetitive information and low discriminative information. Existing methods often use fixed mapping methods to reduce the dimensionality of the fused features, making it difficult to effectively filter based on the correlation and classification contribution of character identity features and scene environment features across different channels. This can easily result in the loss of features with classification value during compression, thus affecting the accuracy and stability of image classification results. Summary of the Invention

[0005] The purpose of this invention is to provide an image classification method and device based on neural networks, thereby solving the problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an image classification method based on neural networks, comprising: S1, acquiring a scene image, and using a convolutional neural network to perform cross-scale adaptive weighted fusion based on the response differences between candidate regions for people and background regions to obtain an initial feature map containing spatial distribution information; S2, based on the initial feature map containing spatial distribution information, using a region proposal network to perform person region localization and bounding box regression processing to obtain local receptive features containing person bounding boxes; S3, if the confidence of the local receptive features containing person bounding boxes is greater than a preset confidence threshold, then performing affine transformation and cropping operations on the local receptive features containing person bounding boxes to obtain aligned person identity features; S4, for the aligned person identity features, adaptively adjusting the attenuation gradient of the identity response in spatial location. The steps are as follows: S5. Determine the character region and boundary weights to obtain a mask weight map for distinguishing semantic level differences; S6. Obtain the mask weight map for distinguishing semantic level differences and the initial feature map containing spatial distribution information. Remove the character region from the initial feature map containing spatial distribution information through matrix multiplication to obtain scene environment features; S7. Based on the scene environment features and the aligned character identity features, perform global aggregation processing through feature concatenation to obtain a fusion feature expression that eliminates competition interference; S8. For the fusion feature expression that eliminates competition interference, perform adaptive compression based on the collaborative discriminative gain of character identity features and scene environment features in each channel to obtain dimensionality-reduced discriminative features; S9. Based on the dimensionality-reduced discriminative features, use a fully connected layer for classification mapping processing to determine the target classification result corresponding to the scene image.

[0007] Preferably, step S1 includes processing the scene image through a convolutional neural network to obtain an initial pixel matrix, obtaining a first response value of the candidate human region and a second response value of the background region in the initial pixel matrix; calculating the response difference based on the first response value and the second response value, and if the response difference is greater than a threshold, determining multiple cross-scale layers; and performing cross-scale adaptive weighted fusion on the multiple cross-scale layers through adaptive weights to obtain an initial feature map containing spatial distribution information.

[0008] Preferably, step S2 includes acquiring an initial feature map containing spatial distribution information, generating an initial set of anchor boxes based on the initial feature map and determining positive sample anchor boxes; performing bounding box regression calculation on the positive sample anchor boxes to obtain bounding box regression offset data; correcting the positive sample anchor boxes using the bounding box regression offset data to obtain a set of proposed character regions; and performing pooling processing on the initial feature map based on the set of proposed character regions to obtain local receptive features containing character bounding boxes.

[0009] Preferably, step S3 includes acquiring local receptive features containing the bounding box of the person and the confidence level corresponding to the local receptive features; if the confidence level is greater than a preset confidence threshold, then determining an affine transformation matrix based on the coordinates of the bounding box of the person; mapping the local receptive features to spatial coordinates through the affine transformation matrix to obtain the mapped feature pixel distribution; determining the cropping operation area in the mapped feature pixel distribution and performing the cropping operation to obtain the aligned person identity features.

[0010] Preferably, step S4 includes obtaining aligned character identity features, extracting the identity responses of the character identity features to obtain a response distribution matrix; calculating the numerical difference based on the response distribution matrix to obtain a set of decay gradients for the identity responses; determining the region division result based on the set of decay gradients; using the region division result to determine the character region and assign boundary weights to obtain semantic hierarchical features with boundary weights; and distinguishing hierarchical differences based on the semantic hierarchical features to obtain a mask weight map used to distinguish semantic hierarchical differences.

[0011] Preferably, step S5 includes obtaining a mask weight map and an initial feature map containing spatial distribution information; processing the mask weight map through feature dimension transformation to obtain an alignment weight matrix; extracting pixel values ​​of the person region based on the alignment weight matrix to generate an inverse mask matrix; if the element value of the inverse mask matrix is ​​greater than a preset threshold, determining that the region corresponding to the element value is a background pixel retention region to obtain a background mask map; performing matrix multiplication between the background mask map and the initial feature map to remove the person region in the initial feature map and obtain scene environment features.

[0012] Preferably, step S6 includes acquiring scene environment features and character identity features; performing dimensional alignment processing on the scene environment features and character identity features to obtain aligned features; performing semantic mapping processing on the aligned features to obtain an associated feature matrix; performing interference filtering processing on the associated feature matrix to determine a pure feature vector, wherein the pure feature vector is generated by adaptive pooling operation; and performing feature concatenation and global aggregation processing on the pure feature vector to obtain a fused feature expression that eliminates competition interference.

[0013] Preferably, step S7 includes acquiring a fusion feature representation that eliminates competitive interference; extracting character identity features and scene environment features from the fusion feature representation and calculating the initial activation intensity; calculating the collaborative discrimination gain based on the initial activation intensity; if the collaborative discrimination gain is greater than a preset threshold, obtaining the first channel feature; if the collaborative discrimination gain is less than or equal to the preset threshold, obtaining the second channel feature; and adaptively compressing the first channel feature and the second channel feature based on the collaborative discrimination gain to obtain the dimensionality-reduced discrimination features.

[0014] Preferably, step S8 includes obtaining the dimensionality-reduced discriminant features, performing a linear transformation on the dimensionality-reduced discriminant features through a mapping matrix to obtain an initial output vector; performing nonlinear mapping and bias compensation on the initial output vector to determine the node weight vector; using a fully connected layer to perform classification mapping on the node weight vector to obtain a category probability distribution; if the maximum probability value in the category probability distribution is greater than a preset threshold, then determining the target classification result corresponding to the scene image based on the maximum probability value.

[0015] A neural network-based image classification device includes: a memory and a processor; The memory is used to store programs; The processor is used to execute the program in the memory to implement the steps of the neural network-based image classification method.

[0016] As can be seen from the above technical solution, the present invention has the following beneficial effects: This neural network-based image classification method and device utilizes a convolutional neural network to perform cross-scale adaptive weighted fusion based on the response differences between candidate human regions and background regions. This allows the initial feature map to consider both local details of the human figure and the overall semantics of the scene, reducing information imbalance between features at different scales. A region proposal network is used to locate the human region and regress its bounding box. Affine transformations and cropping are applied to local receptive features that meet the confidence requirements, which helps reduce the impact of variations in human pose, size, shooting angle, and occlusion on identity feature extraction. The human region and boundary weights are determined by the decay gradient of the identity response in spatial location, and the human region is removed from the initial feature map using a mask weight map. This effectively separates human identity features from scene environment features, reducing coupling and interference between features at different semantic levels. Furthermore, by globally aggregating human identity features and scene environment features and adaptively compressing them based on the collaborative discriminative gain of the two types of features across channels, features with high classification contributions are retained while suppressing redundant information. This improves the accuracy, stability, and discriminative power of classifying complex scene images containing human figures. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a timing diagram of the present invention on the terminal and server; Figure 3 This is a flowchart of the process framework of the present invention; Figure 4 This is a comparison chart of the training convergence curves of this invention; Figure 5 This is a comparison diagram showing the robustness of the alignment of human identity features under different occlusion ratios in this invention. Figure 6 This is a comparison chart of attention heatmaps for character-scene decoupling in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 and Figure 6 As shown, the present invention provides a technical solution: an image classification method based on neural networks, comprising: S1. Acquire scene images and perform cross-scale adaptive weighted fusion based on the response differences between candidate regions of people and background regions using a convolutional neural network to obtain an initial feature map containing spatial distribution information. S2. Based on the initial feature map containing spatial distribution information, a region proposal network is used to perform person region localization and bounding box regression to obtain local receptive features containing person bounding boxes. S3. If the confidence level of the local receptive features containing the bounding box of the person is greater than the preset confidence level threshold, then perform affine transformation and cropping operations on the local receptive features containing the bounding box of the person to obtain the aligned person identity features. S4. For the aligned identity features, the weights of the character region and boundary are adaptively determined based on the decay gradient of the identity response in the spatial location, and a mask weight map is obtained to distinguish semantic level differences. S5. Obtain the mask weight map used to distinguish semantic level differences and the initial feature map containing spatial distribution information. Remove the person region in the initial feature map containing spatial distribution information through matrix multiplication to obtain scene environment features. S6. Based on the scene environment features and the aligned character identity features, perform global aggregation processing through feature splicing operations to obtain a fused feature expression that eliminates competition interference. S7. To eliminate competition interference, the fusion feature representation is adaptively compressed based on the collaborative discriminative gain of the character identity feature and the scene environment feature in each channel to obtain the dimensionality-reduced discriminative features. S8. Based on the dimensionality-reduced discriminative features, a fully connected layer is used for classification mapping to determine the target classification result corresponding to the scene image.

[0020] In this embodiment, the scene image to be classified is first acquired. The scene image can be provided by a surveillance camera, a mobile terminal, a vehicle-mounted camera, or an image database. The width and height of the scene image are adjusted to 640 pixels. The red, green, and blue channels of the scene image are then normalized to ensure that the pixel distribution of the input image is consistent with the pixel distribution used in the neural network training phase.

[0021] Furthermore, the preprocessed scene image is input into a convolutional neural network, which performs convolution processing on the scene image in order from shallow to deep layers, and outputs feature maps at three scales respectively.

[0022] The feature map at the first scale has a width of 160 feature units, a height of 160 feature units, and 256 channels; the feature map at the second scale has a width of 80 feature units, a height of 80 feature units, and 256 channels; and the feature map at the third scale has a width of 40 feature units, a height of 40 feature units, and 256 channels.

[0023] Specifically, a convolutional process with a kernel width of 3 feature units, a kernel height of 3 feature units, and a stride of 2 feature units is used to adjust the feature map of the first scale to a width of 80 feature units, a height of 80 feature units, and 256 channels.

[0024] Bilinear interpolation was used to adjust the feature map of the third scale to a width of 80 feature units, a height of 80 feature units, and 256 channels.

[0025] The feature map at the second scale retains a width of 80 feature units, a height of 80 feature units, and a channel count of 256. After the above processing, the feature maps at the three scales have the same spatial dimensions and the same number of channels.

[0026] In addition, convolution processing with a kernel width of 1 feature unit and a kernel height of 1 feature unit was performed on the feature maps of the three scales respectively, compressing the 256 channel responses corresponding to each spatial location into a single character response value; then, numerical compression processing was used to limit each character response value to between 0 and 1, resulting in character response maps corresponding to the three scales respectively. Specifically, for each character response map, spatial locations with response values ​​greater than or equal to 0.50 are defined as candidate character regions, while spatial locations with response values ​​less than 0.50 are defined as background regions.

[0027] For each scale of the feature map, the feature responses of all spatial locations within the candidate person region are extracted, and the arithmetic mean of the above feature responses is calculated. Then, the feature responses of all spatial locations within the background region are extracted, and the arithmetic mean of the above feature responses is calculated. The arithmetic mean of the candidate person region is subtracted from the arithmetic mean of the background region to obtain the response difference between the person region and the background region corresponding to that scale.

[0028] The response differences corresponding to the three scales are exponentially calculated separately. Each exponential calculation result is divided by the sum of the three exponential calculation results to obtain the fusion weights corresponding to the three scales respectively. The sum of the values ​​of the three fusion weights is 1.

[0029] Specifically, when the response difference between the candidate area of ​​the person and the background area increases, the fusion weight of the corresponding scale increases accordingly; when the response difference between the candidate area of ​​the person and the background area decreases, the fusion weight of the corresponding scale decreases accordingly.

[0030] Specifically, each feature value in the feature map of the first scale is multiplied by the fusion weight corresponding to the first scale, each feature value in the feature map of the second scale is multiplied by the fusion weight corresponding to the second scale, and each feature value in the feature map of the third scale is multiplied by the fusion weight corresponding to the third scale.

[0031] The feature values ​​of the three weighted feature maps at the same spatial location and the same channel location are added together to obtain an initial feature map containing spatial distribution information; the width of the initial feature map is 80 feature units, the height is 80 feature units, and the number of channels is 256.

[0032] Specifically, the initial feature map simultaneously includes the person's outline, position, background texture, object structure, and scene category semantics. By determining the fusion weights based on the response difference between the candidate region of the person and the background region, the three scales can participate in feature fusion according to the distribution of the person's response in the current scene image.

[0033] An initial feature map containing spatial distribution information is input into a region proposal network; the region proposal network consists of a shared convolutional layer, a person confidence prediction branch, and a bounding box regression branch.

[0034] The shared convolutional layer uses a convolutional kernel with a width of 3 feature units and a height of 3 feature units to perform position-by-position scanning processing on the initial feature map.

[0035] Nine candidate bounding boxes are set at each spatial location in the initial feature map; the nine candidate bounding boxes are formed by a combination of three sets of sizes and three sets of aspect ratios.

[0036] Specifically, the three sizes are 32 pixels, 64 pixels, and 128 pixels, respectively; the three aspect ratios are equal width and height, half the height, and twice the height, respectively.

[0037] The person confidence prediction branch classifies each person candidate box and outputs the confidence score that the candidate box contains the person target. The confidence score ranges from 0 to 1.

[0038] Furthermore, the bounding box regression branch outputs four adjustment parameters for each person candidate box; the four adjustment parameters are used to adjust the horizontal position of the person candidate box center, the vertical position of the person candidate box center, the width of the person candidate box, and the height of the person candidate box.

[0039] Specifically, when adjusting the center position of the candidate character frame, the horizontal adjustment parameter is multiplied by the original width of the candidate character frame, and then the result of the multiplication is added to the horizontal coordinates of the center of the original candidate character frame to obtain the horizontal coordinates of the center of the adjusted candidate character frame.

[0040] Multiply the vertical adjustment parameter by the original height of the candidate character frame, and then add the result of the multiplication to the vertical coordinates of the center of the original candidate character frame to obtain the vertical coordinates of the center of the adjusted character bounding box.

[0041] When adjusting the width of the candidate character frame, the width adjustment parameter is exponentially calculated, and the result of the exponential calculation is multiplied by the original width of the candidate character frame to obtain the width of the adjusted character bounding box.

[0042] When adjusting the height of the candidate character frame, the height adjustment parameter is exponentially calculated, and the result of the exponential calculation is multiplied by the original height of the candidate character frame to obtain the height of the adjusted character bounding box.

[0043] Non-maximum suppression is applied to the adjusted character bounding boxes; the intersection area between any two character bounding boxes is calculated, and the total coverage area of ​​the two bounding boxes is also calculated. The intersection-union ratio (IUR) between the two bounding boxes is obtained by dividing the intersection area by the total coverage area.

[0044] When the intersection-union ratio between two bounding boxes is greater than 0.50, retain the bounding box with the higher confidence score and delete the bounding box with the lower confidence score.

[0045] Furthermore, after performing nonmaximum suppression, the bounding boxes of the characters are arranged in descending order of their confidence levels, and the bounding boxes of the top 100 characters are retained.

[0046] Based on the preserved character bounding boxes, region of interest alignment is performed on the initial feature map; 14 rows and 14 columns of sampling positions are set inside each character bounding box, for a total of 196 sampling positions.

[0047] For each sampling location, bilinear interpolation is performed on the feature values ​​of the four surrounding feature locations based on the distance between the sampling location and the surrounding four feature locations to obtain the feature value corresponding to the sampling location. After the region of interest alignment processing, the height of the local receptive feature corresponding to each person bounding box is 14 feature units, the width is 14 feature units, and the number of channels is 256.

[0048] The confidence threshold for the person is set to 0.70. When the confidence of the person corresponding to the person bounding box is greater than or equal to 0.70, the person bounding box and its local perceptual features are determined as the effective person region and effective local perceptual features.

[0049] When the confidence score of the person corresponding to the bounding box is less than 0.70, delete the bounding box and its local receptive features, and do not input them into the subsequent processing.

[0050] For each valid local perceptual feature, the center of the character bounding box is used as the transformation center, and affine transformation parameters are generated based on the width, height, and position of the character bounding box.

[0051] The affine transformation maps the top left corner of the character bounding box to the top left corner of the standard feature region, the top right corner of the character bounding box to the top right corner of the standard feature region, and the bottom left corner of the character bounding box to the bottom left corner of the standard feature region.

[0052] The height of the standard feature region is set to 28 feature units, and the width is set to 14 feature units.

[0053] Specifically, based on the affine transformation parameters, 28 rows and 14 columns of sampling positions are generated in the effective local receptive features; for each sampling position, bilinear interpolation is used to obtain the feature values ​​corresponding to 256 channels, thus obtaining the aligned human identity features.

[0054] Each aligned character identity feature has a height of 28 feature units, a width of 14 feature units, and 256 channels.

[0055] The process involves performing convolution processing on the character's identity features, with a kernel width of 1 feature unit and a kernel height of 1 feature unit, mapping the 256 channel responses of each spatial location to an identity response value; then, each identity response value is limited to between 0 and 1 to obtain the character identity response map.

[0056] The height of the character identity response image is 28 response units, and the width is 14 response units.

[0057] Furthermore, for each spatial location in the character identity response map, the identity response values ​​of the adjacent locations to the left and right are obtained respectively; the identity response value of the adjacent location to the right is subtracted from the identity response value of the adjacent location to the left, the absolute value of the difference is taken, and then the absolute value is divided by 2 to obtain the change in response in the horizontal direction of the spatial location.

[0058] Obtain the identity response values ​​of the adjacent positions above and below the spatial location respectively; subtract the identity response value of the adjacent position above from the identity response value of the adjacent position below, take the absolute value of the difference, and then divide the absolute value by 2 to obtain the change in response of the spatial location in the vertical direction.

[0059] Specifically, for spatial locations located at the edge of the character identity response map, when there is no adjacent location in the corresponding direction, the identity response value of the edge location itself is used to replace the identity response value of the missing adjacent location.

[0060] Multiply the change in horizontal response by itself, multiply the change in vertical response by itself, add the two multiplication results, and then perform a square root operation on the sum to obtain the identity response attenuation gradient corresponding to that spatial location.

[0061] Based on the identity response value and identity response attenuation gradient of each spatial location, the person's identity response map is divided into regions.

[0062] Specifically, when the identity response value of a spatial location is greater than or equal to 0.60, and the identity response attenuation gradient of that spatial location is less than or equal to 0.10, the spatial location is determined to be the internal region of the character.

[0063] When the identity response value of a spatial location is greater than 0.30, and the spatial location does not meet the determination criteria for the internal region of a character, the spatial location is determined as the boundary region of the character.

[0064] When the identity response value of a spatial location is less than or equal to 0.30, the spatial location is identified as a background area.

[0065] A mask weight map is generated based on the character's internal area, character boundary area, and background area.

[0066] For the area inside the character, set the mask weight of the corresponding spatial location to 0.

[0067] For the background region, the mask weight at the corresponding spatial location is set to 1.

[0068] For the boundary area of ​​the character, first subtract the identity response value of the spatial location from 0.60, and then divide the difference by 0.30; if the calculation result is less than 0, the mask weight is set to 0; if the calculation result is greater than 1, the mask weight is set to 1; if the calculation result is between 0 and 1, the calculation result is directly determined as the mask weight.

[0069] Therefore, the mask weight corresponding to the internal area of ​​the character is fixed at 0, the mask weight corresponding to the background area is fixed at 1, and the mask weight corresponding to the boundary area of ​​the character changes continuously according to the identity response value.

[0070] Furthermore, the mask weight map with a height of 28 weight units and a width of 14 weight units is mapped to the corresponding position of the initial feature map according to the inverse mapping relationship of the affine transformation process.

[0071] Specifically, bilinear interpolation is used to adjust the mapped mask weight map to a height of 80 weight units and a width of 80 weight units, so that the mask weight map has the same spatial size as the initial feature map.

[0072] When there is only one bounding box for a person in the scene image with a confidence score greater than or equal to 0.70, the mask weight map corresponding to that person bounding box is determined as the scene-level mask weight map.

[0073] When there are two or more bounding boxes of people in the scene image with a confidence score greater than or equal to 0.70, generate mask weight maps corresponding to the bounding boxes of the people respectively.

[0074] For each spatial location, the weights of all character mask weight maps at that spatial location are multiplied sequentially, and the result of the multiplication is determined as the scene-level mask weight corresponding to that spatial location, thus forming a scene-level mask weight map.

[0075] The scene-level mask weight map is copied to 256 channels, making its number of channels the same as the number of channels in the initial feature map.

[0076] For each spatial location and each channel in the initial feature map, the corresponding feature value is multiplied by the scene-level mask weight to obtain the scene environment features.

[0077] When a spatial location belongs to the interior region of a person, the mask weight corresponding to that location is 0, and the feature values ​​of all 256 channels of the initial feature map at that location are set to 0.

[0078] When a spatial location belongs to the background region, the mask weight corresponding to that location is 1, and the 256 channel feature values ​​of the initial feature map at that location remain unchanged.

[0079] When a spatial location belongs to the boundary region of a person, the 256 channel feature values ​​of the initial feature map at that location are scaled according to a mask weight between 0 and 1.

[0080] Specifically, global average pooling is performed on the scene environment features; for each channel, all feature values ​​of that channel in the 80 rows and 80 columns are summed, and the sum is divided by 6400 to obtain the scene environment feature value corresponding to that channel; the above processing is performed on each of the 256 channels to obtain a scene environment feature vector containing 256 values.

[0081] Specifically, global average pooling is performed on each aligned character identity feature; for each channel, all feature values ​​in the 28th row and 14th column spatial position of that channel are summed, and the sum is divided by 392 to obtain the character identity feature value corresponding to that channel; the above processing is performed on each of the 256 channels to obtain a character identity feature vector containing 256 values.

[0082] When there is only one bounding box for a person in the scene image with a confidence score greater than or equal to 0.70, the person identity feature vector corresponding to that person is determined as the scene-level person identity feature vector.

[0083] When there are two or more bounding boxes for people in the scene image with a confidence score greater than or equal to 0.70, the confidence scores of all the bounding boxes are summed.

[0084] For each character, divide the character confidence score of that character by the sum of the confidence scores of all characters to obtain the identity fusion weight for that character.

[0085] Each character's 256 identity feature values ​​are multiplied by the corresponding identity fusion weight, and then the weighted identity feature values ​​of all characters at the same channel position are summed to obtain the scene-level character identity feature vector.

[0086] By concatenating the scene-level character identity feature vector containing 256 values ​​with the scene environment feature vector containing 256 values, a fused feature representation containing 512 values ​​is obtained.

[0087] The fusion feature representation containing 512 values ​​is input into the channel collaborative analysis module.

[0088] The channel collaborative analysis module consists of a first fully connected layer, a linear rectification activation layer, a second fully connected layer, and a gating layer.

[0089] The first fully connected layer transforms the fused feature representation containing 512 values ​​into an intermediate feature containing 128 values.

[0090] The linear rectification activation layer evaluates each value in the intermediate features; when the value is greater than 0, it remains unchanged; when the value is less than or equal to 0, it is set to 0.

[0091] The second fully connected layer transforms the intermediate features, which contain 128 values, into channel control parameters, which contain 512 values.

[0092] The gating layer performs numerical compression on the 512 channel control parameters, converting each channel control parameter into a collaborative discriminative gain between 0 and 1.

[0093] For each channel in the fusion feature representation, the feature value of that channel is multiplied by the corresponding co-discriminative gain to obtain the weighted fusion feature.

[0094] When the person's identity information and the scene environment information produce consistent discriminative effects on the same classification result, the corresponding channel obtains a co-discriminative gain greater than 0.50 during training; when the person's identity information and the scene environment information produce opposite discriminative effects on the same classification result, the corresponding channel obtains a co-discriminative gain less than or equal to 0.50 during training.

[0095] The weighted fusion feature containing 512 values ​​is input into the dimension reduction fully connected layer; the dimension reduction fully connected layer transforms the weighted fusion feature containing 512 values ​​into a feature containing 256 values.

[0096] Specifically, the feature containing 256 values ​​is subjected to batch normalization, linear rectification activation, and random deactivation in sequence.

[0097] Batch normalization is used to adjust the values ​​of each feature channel within the same training batch to a uniform distribution range.

[0098] Linear rectification activation is used to set eigenvalues ​​less than or equal to 0 to 0, while keeping eigenvalues ​​greater than 0 unchanged.

[0099] The random deactivation ratio is set to 0.50. Specifically, during training, 128 feature values ​​are randomly selected from 256 feature values ​​and set to 0, while the remaining 128 feature values ​​are amplified according to the compensation ratio. After the above processing is completed, a discriminative feature containing 256 values ​​is obtained.

[0100] Furthermore, the discriminative features are input into the classification fully connected layer. The number of outputs of the classification fully connected layer is the same as the number of categories in the target classification task, and each output value corresponds to the classification score of a candidate category.

[0101] Perform exponential operations on all category scores separately, and divide each exponential operation result by the sum of all exponential operation results to obtain the category probability corresponding to each candidate category; the sum of all category probabilities is 1.

[0102] Compare the class probabilities of all candidate classes, and determine the candidate class with the highest class probability value as the target classification result for the scene image.

[0103] During neural network training, training images containing both scene category labels and person bounding box labels are used.

[0104] Specifically, the training loss includes scene classification loss, region proposal network character classification loss, and character bounding box regression loss.

[0105] Scene classification loss is used to calculate the difference between the predicted scene category and the true scene category.

[0106] The region proposal network person classification loss is used to calculate the difference between the predicted person candidate box and the actual person candidate box annotation.

[0107] The Person Bounding Box Regression Loss is used to calculate the difference between the location parameters of the predicted person bounding box and the location parameters of the actual person bounding box.

[0108] Specifically, the scene classification loss, the region proposal network character classification loss, and the character bounding box regression loss are added together to obtain the total training loss of the neural network.

[0109] The training batch size is set to 16, meaning that 16 training images are input for each parameter update.

[0110] The initial learning rate was set to 0.0001, the weight decay coefficient was set to 0.0001, and the number of training rounds was set to 100.

[0111] After every 10 rounds of training, multiply the current learning rate by 0.8, and use the result as the learning rate for the next 10 rounds of training.

[0112] After each training round, the scene classification accuracy is calculated using the verification image. After 100 training rounds, the network parameters corresponding to the round with the highest verification image classification accuracy are saved, and the saved network parameters are used to perform scene image classification.

[0113] S1 includes processing the scene image through a convolutional neural network to obtain an initial pixel matrix, obtaining the first response value of the candidate human region and the second response value of the background region in the initial pixel matrix; calculating the response difference based on the first response value and the second response value, and determining multiple cross-scale layers if the response difference is greater than a threshold; performing cross-scale adaptive weighted fusion on the multiple cross-scale layers through adaptive weights to obtain an initial feature map containing spatial distribution information.

[0114] In this embodiment, the acquired scene image is uniformly adjusted to a three-channel image with a width of 640 pixels and a height of 640 pixels, and the pixel values ​​in each channel that range from 0 to 255 are divided by 255 respectively, so that the processed pixel values ​​are in the range of 0 to 1.

[0115] The scene image that has undergone normalization is input into the convolutional neural network, which then performs convolution, normalization, activation, and downsampling processes sequentially.

[0116] Specifically, the input convolutional layer of the convolutional neural network uses a convolutional kernel with a size of 7 pixels by 7 pixels and a stride of 2 pixels to convert the scene image into a convolutional feature with a width of 320 feature units, a height of 320 feature units, and 64 channels. Then, max pooling with a size of 3 feature units by 3 feature units and a stride of 2 feature units is used to convert the convolutional feature into an initial pixel matrix with a width of 160 feature units, a height of 160 feature units, and 64 channels. Each spatial position in the initial pixel matrix corresponds to a 4-pixel by 4-pixel region in the scene image, and the 64 channels record the feature response formed by the convolution operation in that region.

[0117] Furthermore, a character response extraction convolutional layer is set after the initial pixel matrix.

[0118] Specifically, the character response extraction convolutional layer uses a convolutional kernel with a size of 1 feature unit multiplied by 1 feature unit. It performs a weighted summation of the 64 channel feature values ​​corresponding to each spatial location and adds the bias value corresponding to the convolutional layer to obtain the original character response value at that spatial location. The original character response value is then processed by a logistic function, which first calculates the natural exponent value corresponding to the negative of the original character response value, then adds the natural exponent value to 1, and finally divides 1 by the above summation result to obtain the character response value between 0 and 1. All character response values ​​are arranged according to the row and column positions of the initial pixel matrix to form a character response matrix.

[0119] In this embodiment, during the training phase, candidate regions for people are determined by using bounding box annotations of people in the training images.

[0120] Specifically, the x and y coordinates of the person's bounding box are divided by 4, and the resulting coordinates are mapped to the initial pixel matrix. The spatial positions inside the person's bounding box after mapping are included in the person's candidate region, and the spatial positions outside the person's bounding box and separated from the edge of the person's bounding box by more than 2 feature units are included in the background region. The interval of 2 feature units is set to exclude the mixed response at the junction of the person's outline and the background, and to prevent the boundary position from participating in the calculation of both the person's response and the background response at the same time.

[0121] In addition, since there are no manually labeled character bounding boxes during the reasoning phase, the candidate regions for characters are determined based on the character response matrix.

[0122] Specifically, spatial locations with a character response value greater than or equal to 0.50 are marked as candidate locations. Candidate locations that are adjacent in the horizontal, vertical, or diagonal directions are connected and merged to form connected regions. Connected regions containing less than 16 spatial locations are deleted, and the remaining connected regions are determined as character candidate regions. Spatial locations with a character response value less than 0.50 and separated from the edge of the character candidate region by more than 2 feature units are determined as background regions.

[0123] As a preferred embodiment, the character response judgment value of 0.50 is fixed after the neural network training is completed.

[0124] When the convolutional layer for human response extraction is trained using cross-entropy loss, the target value of the human annotation region is set to 1, and the target value of the background annotation region is set to 0. The output value of the logistic function, 0.50, corresponds to the equal probability boundary between the human target and the background target. Therefore, 0.50 is used as the judgment value for the candidate human position and the background position, and this judgment value is no longer changed according to the image to be classified.

[0125] Furthermore, after determining the candidate character region, the character response value of each spatial location within the candidate character region is read. All read character response values ​​are added together sequentially, and the sum is divided by the number of spatial locations contained in the candidate character region. The resulting value is used as the first response value.

[0126] When two candidate regions for a person are formed in the scene image, the average response of the two candidate regions is calculated first, and then they are weighted and merged according to the number of spatial locations contained in each candidate region.

[0127] Specifically, the average response of each candidate region is multiplied by the number of spatial locations contained in that region, the products are added together, and then divided by the sum of the number of spatial locations contained in all candidate regions. The resulting value is used as the first response value.

[0128] Furthermore, the character response value of each spatial location within the background area is read, all read character response values ​​are added together sequentially, and the sum is divided by the number of spatial locations contained in the background area. The resulting value is used as the second response value. The first response value represents the response intensity of the candidate character region to the character semantics, and the second response value represents the response intensity of the background region to the character semantics.

[0129] Specifically, the difference between the first response value and the second response value is taken as the response difference.

[0130] When the response difference is positive, it indicates that the average response of the candidate character area exceeds that of the background area; when the response difference is 0, it indicates that the average response of the candidate character area and the background area are the same; when the response difference is negative, it indicates that the current candidate character area has not formed an effective distinction in character response.

[0131] In this embodiment, the threshold corresponding to the response difference is determined using the verification image after training is completed and remains unchanged during the image classification stage.

[0132] Specifically, the threshold determination process is as follows: input the verification images one by one into the trained convolutional neural network, and calculate the response difference of each verification image according to the aforementioned steps; classify the verification images with an intersection-union area ratio (IUAR) of the candidate region and the manually labeled bounding box of the person into the effective response group, and classify the verification images with an IUAR of less than 0.50 into the invalid response group; the IUAR calculation process is as follows: first calculate the area of ​​the overlapping part between the candidate region and the manually labeled bounding box of the person, then calculate the sum of the areas of the candidate region and the manually labeled bounding box of the person, then subtract the area of ​​the overlapping part from the sum of the areas, and finally divide the area of ​​the overlapping part by the area after subtraction.

[0133] Furthermore, the average effective response is obtained by summing all response differences within the effective response group and dividing by the number of images in the effective response group; the average invalid response is obtained by summing all response differences within the invalid response group and dividing by the number of images in the invalid response group; the average effective response and the average invalid response are added together, and the result is divided by 2 to determine the response difference threshold.

[0134] The average effective response of the verification image used in this embodiment is 0.22 and the average invalid response is 0.08. The sum of the two is 0.30, which is then divided by 2 to determine the response difference threshold as 0.15.

[0135] When the response difference is greater than 0.15, the convolutional neural network determines the first cross-scale layer, the second cross-scale layer, and the third cross-scale layer.

[0136] Specifically, the first cross-scale layer is taken from the output of the first feature extraction stage of the convolutional neural network, with a width of 160 feature units, a height of 160 feature units, and 256 channels; the second cross-scale layer is taken from the output of the second feature extraction stage, with a width of 80 feature units, a height of 80 feature units, and 512 channels; the third cross-scale layer is taken from the output of the third feature extraction stage, with a width of 40 feature units, a height of 40 feature units, and 1024 channels; the three cross-scale layers record the feature responses of the scene image at sampling intervals of 4 pixels, 8 pixels, and 16 pixels, respectively.

[0137] In addition, in order to enable the three cross-scale layers to be integrated in the same spatial location, the channels and sizes of each cross-scale layer are first adjusted.

[0138] Specifically, after the first cross-scale layer is processed by convolution with a size of 1 feature unit by 1 feature unit, the number of channels remains at 256. Then, a convolution with a size of 3 feature units by 3 feature units and a stride of 2 feature units is used to adjust the width and height from 160 feature units to 80 feature units.

[0139] The second cross-scale layer uses a convolutional kernel with a size of 1 feature unit by 1 feature unit to convert the 512 channel feature values ​​at each spatial location into 256 channel feature values, while keeping its width and height unchanged at 80 feature units.

[0140] The third cross-scale layer uses a convolutional kernel with a size of 1 feature unit by 1 feature unit to convert the 1024 channel feature values ​​at each spatial location into 256 channel feature values. Then, bilinear interpolation is used to adjust the width and height from 40 feature units to 80 feature units each. The feature value at the interpolation position is determined based on the distance to the four known feature positions around that position. The closer the known feature position is to the interpolation position, the larger the interpolation coefficient is obtained. The sum of the four interpolation coefficients is 1.

[0141] After the above processing is completed, the first, second, and third cross-scale layers each have 80 rows, 80 columns, and 256 channels.

[0142] Specifically, the candidate regions for people and the background regions in the initial pixel matrix are mapped to three cross-scale layers according to their coordinate ratios. For the first cross-scale layer, the horizontal and vertical coordinates of the candidate regions in the initial pixel matrix are divided by 2 to obtain the corresponding positions of the candidate regions in the size adjustment results of the first cross-scale layer. For the second cross-scale layer, the coordinate positions of the candidate regions are directly used. For the third cross-scale layer, the coordinates of the candidate regions in the original third cross-scale layer are magnified by 2 times to obtain the positions of the candidate regions after size adjustment. The background region is mapped according to the same coordinate transformation relationship.

[0143] Specifically, within the first cross-scale layer, all feature values ​​in the mapped candidate person region are summed and then divided by the product of the number of spatial locations contained in the candidate person region and 256 channels to obtain the average response of the person region in the first cross-scale layer; all feature values ​​in the mapped background region are summed and then divided by the product of the number of spatial locations contained in the background region and 256 channels to obtain the average response of the background region in the first cross-scale layer; the average response of the person region is subtracted from the average response of the background region to obtain the scale response difference of the first cross-scale layer.

[0144] Following the same calculation process, the scale response differences of the second and third cross-scale layers were obtained respectively.

[0145] When any scale response difference is less than 0, the scale response difference is set to 0; when any scale response difference is equal to or greater than 0, its original value is retained; then the three processed scale response differences are added together to obtain the total scale response value.

[0146] Specifically, when the total scale response value is greater than 0, the scale response difference of the first cross-scale layer is divided by the total scale response value to obtain the first adaptive weight; the scale response difference of the second cross-scale layer is divided by the total scale response value to obtain the second adaptive weight; the scale response difference of the third cross-scale layer is divided by the total scale response value to obtain the third adaptive weight; all three adaptive weights are between 0 and 1, and their sum is 1.

[0147] When the differences between the three scale responses are all zero after being zeroed out, the first adaptive weight, the second adaptive weight, and the third adaptive weight are all set to one-third, so that the three cross-scale layers participate in the fusion in the same proportion, thus avoiding the calculation result of division by zero when the total scale response value is 0.

[0148] Furthermore, the feature values ​​of each spatial location and each channel in the first cross-scale layer after size adjustment are multiplied by the first adaptive weight; the feature values ​​of each spatial location and each channel in the second cross-scale layer are multiplied by the second adaptive weight; the feature values ​​of each spatial location and each channel in the third cross-scale layer after size adjustment are multiplied by the third adaptive weight; and then the weighted feature values ​​of the three cross-scale layers at the same row position, the same column position, and the same channel position are added together to form a fused feature map with a width of 80 feature units, a height of 80 feature units, and 256 channels.

[0149] Specifically, the fused feature maps are integrated through a convolutional layer with a size of 3 feature units by 3 feature units and a stride of 1 feature unit.

[0150] Before convolution, one row or column of feature units with a value of 0 is added to the top, bottom, left and right sides of the fused feature map to keep the width and height of the convolutional layer at 80 feature units. The 256 channels output by the convolutional layer record the fusion response of the character outline, character region position, background texture, scene object structure and scene semantics in a unified spatial coordinate system. This yields an initial feature map containing spatial distribution information, which is then input into the region proposal network for character region localization and bounding box regression.

[0151] S2 includes obtaining an initial feature map containing spatial distribution information, generating an initial set of anchor boxes based on the initial feature map and determining positive sample anchor boxes; performing bounding box regression calculation on the positive sample anchor boxes to obtain bounding box regression offset data; correcting the positive sample anchor boxes using the bounding box regression offset data to obtain a set of proposed character regions; and performing pooling processing on the initial feature map based on the set of proposed character regions to obtain local receptive features containing character bounding boxes.

[0152] In this embodiment, an initial feature map is obtained by cross-scale adaptive weighted fusion.

[0153] Specifically, the initial feature map contains 80 rows, 80 columns, and 256 feature channels. One spatial location in the initial feature map corresponds to an 8-pixel by 8-pixel region in the scene image. Each spatial location preserves the outline of the person, the structure of the person's limbs, the background texture, and their positional relationship in the scene image, providing a feature basis for the localization of the person's region.

[0154] Furthermore, the region proposal network uses each spatial location in the initial feature map as the center of the anchor box.

[0155] In this context, the distance between the centers of adjacent anchor frames is 8 pixels in both the horizontal and vertical directions of the scene image. The center of the anchor frame corresponding to the spatial position in the first row and first column of the initial feature map is set at 4 pixels on the horizontal and 4 pixels on the vertical coordinates of the scene image. Subsequently, for each spatial position moved along the row or column direction, the center of the anchor frame moves 8 pixels in the scene image accordingly.

[0156] Specifically, nine initial anchor frames are set at the center of each anchor frame; the nine initial anchor frames are obtained by combining three reference sizes and three aspect ratios; the three reference sizes correspond to reference side lengths of 32 pixels, 64 pixels and 128 pixels respectively, and the three aspect ratios are 1:1, 1:2 and 2:1 respectively.

[0157] Specifically, for an initial anchor frame with a width-to-height ratio of 1:1, both its width and height are set to the corresponding reference side length; for an initial anchor frame with a width-to-height ratio of 1:2, the height is set to twice the width while keeping the anchor frame area equal to the square of the reference side length; and for an initial anchor frame with a width-to-height ratio of 2:1, the width is set to twice the height while keeping the anchor frame area unchanged.

[0158] Specifically, the initial feature map includes 6,400 spatial locations, and each spatial location generates 9 initial anchor boxes, thus obtaining an initial anchor box set containing 57,600 initial anchor boxes.

[0159] Specifically, for initial anchor frames whose boundaries exceed the scene image range, the left and top boundaries are limited to 0 pixels, and the right and bottom boundaries are limited to 639 pixels, so that all initial anchor frames are within the scene image range.

[0160] Furthermore, during the training phase, positive sample anchor boxes are determined based on the character annotation boxes.

[0161] Specifically, for each initial anchor box, the intersection-union ratio (IUU) between it and the annotation boxes of each person in the scene image is calculated.

[0162] In the specific calculation, first determine the width and height of the overlapping area between the initial anchor frame and the character annotation frame, and multiply the width of the overlapping area by the height to obtain the overlapping area; add the area of ​​the initial anchor frame and the area of ​​the character annotation frame, and then subtract the overlapping area to obtain the union area of ​​the two; divide the overlapping area by the union area, and the resulting value is the intersection-union ratio between the initial anchor frame and the character annotation frame.

[0163] Furthermore, after calculating the intersection-union ratio (IUR) of the same initial anchor frame with each character's bounding box in the scene image, the character bounding box with the largest IUR value is determined as the target character bounding box corresponding to that initial anchor frame.

[0164] When the maximum crossover ratio is greater than or equal to 0.70, the corresponding initial anchor frame is determined as a positive sample anchor frame.

[0165] In addition, to ensure that every person's bounding box participates in bounding box regression training, for each person's bounding box, the initial anchor box with the largest intersection-union ratio with that person's bounding box is directly determined as the positive sample anchor box.

[0166] In this case, even if the maximum crossover ratio is less than 0.70, the initial anchor frame is still used as a positive sample anchor frame.

[0167] As a preferred embodiment, the positive sample determination threshold is determined using validation images outside the training set.

[0168] Specifically, 0.50, 0.60, 0.70, and 0.80 are used as candidate thresholds to determine positive samples for all initial anchor boxes in the verification image. For each candidate threshold, the number of person annotation boxes covered by positive sample anchor boxes is counted, and this number is divided by the total number of person annotation boxes to obtain the person recall rate. At the same time, the number of positive sample anchor boxes with an intersection-union ratio (IU) greater than or equal to 0.50 with person annotation boxes is counted, and this number is divided by the total number of positive sample anchor boxes to obtain the positive sample precision rate.

[0169] The method involves multiplying the person recall rate by the positive sample precision, multiplying the result by 2, and then dividing by the sum of the person recall rate and the positive sample precision to obtain the comprehensive evaluation value corresponding to the candidate threshold.

[0170] In the verification results, the comprehensive evaluation value corresponding to the threshold of 0.70 is the maximum value. Therefore, 0.70 is determined as the positive sample judgment threshold and remains unchanged during the training process.

[0171] Furthermore, after determining the positive sample anchor boxes, the initial feature maps are input into the shared convolutional layers of the region proposal network.

[0172] Specifically, the shared convolutional layer uses a convolutional kernel with 3 feature units multiplied by 3 feature units, a stride of 1 feature unit, and adds 1 row or 1 column of feature units with a value of 0 around the initial feature map, so that the feature map after convolution still maintains 80 rows, 80 columns and 256 channels.

[0173] In this embodiment, the output of the shared convolutional layer is input into the person classification branch and the bounding box regression branch, respectively.

[0174] Specifically, the person classification branch outputs a person confidence score for each initial anchor box, which is used to characterize the probability that a person target exists within that initial anchor box; the bounding box regression branch outputs four bounding box regression offset data for each positive sample anchor box. These four bounding box regression offset data are used to correct the horizontal position of the anchor box center, the vertical position of the anchor box center, the anchor box width, and the anchor box height, respectively.

[0175] Among them, when training the bounding box regression branch, four regression objectives are determined based on the positional relationship between the positive sample anchor box and the corresponding target person box.

[0176] When determining the horizontal regression target, first subtract the horizontal coordinate of the center of the positive sample anchor frame from the horizontal coordinate of the center of the target person frame to obtain the horizontal center distance. Then divide the center distance by the width of the positive sample anchor frame to obtain the horizontal regression target.

[0177] When determining the vertical regression target, first subtract the ordinate of the positive sample anchor frame from the ordinate of the target person frame's center to obtain the vertical center distance. Then divide this center distance by the height of the positive sample anchor frame to obtain the vertical regression target.

[0178] When determining the width regression target, divide the width of the target person frame by the width of the positive sample anchor frame, and perform a natural logarithmic operation on the resulting ratio to obtain the width regression target.

[0179] When determining the height regression target, divide the height of the target person frame by the height of the positive sample anchor frame, and perform a natural logarithmic operation on the resulting ratio to obtain the height regression target.

[0180] Furthermore, after the bounding box regression branch outputs four predicted data points, the corresponding regression target is subtracted from each predicted data point to obtain four regression errors.

[0181] For each regression error, when its absolute value is less than 1, the regression error is multiplied by itself and the result is divided by 2; when its absolute value is greater than or equal to 1, 0.5 is subtracted from the absolute value of the regression error; the four calculation results are added together to obtain the bounding box regression loss corresponding to the positive sample anchor box, and the network parameters of the bounding box regression branch are adjusted according to this loss.

[0182] Furthermore, after completing the training of the person classification branch and the bounding box regression branch, the region proposal network performs person region localization on the scene image to be classified.

[0183] The character classification branch outputs the character confidence score for each initial anchor box in the initial anchor box set, while the bounding box regression branch simultaneously outputs the corresponding four bounding box regression offset data.

[0184] When correcting the horizontal position of the anchor frame center, the first bounding box regression offset data is multiplied by the width of the initial anchor frame, and then the product is added to the horizontal coordinate of the initial anchor frame center to obtain the horizontal coordinate of the corrected bounding box center.

[0185] When correcting the vertical position of the anchor frame center, multiply the second bounding box regression offset data by the initial anchor frame height, and then add the resulting product to the ordinate of the initial anchor frame center to obtain the ordinate of the corrected bounding box center.

[0186] When correcting the bounding box width, the natural exponent is calculated on the third bounding box regression offset data, and then the result is multiplied by the width of the initial anchor box to obtain the corrected bounding box width.

[0187] When correcting the bounding box height, the natural exponent is calculated on the fourth bounding box regression offset data, and then the result is multiplied by the height of the initial anchor box to obtain the corrected bounding box height.

[0188] Furthermore, the four boundaries of the character area proposal frame are determined based on the corrected center coordinates, width, and height.

[0189] Specifically, subtract half the width from the center x-coordinate to get the left boundary; add half the width to the center x-coordinate to get the right boundary; subtract half the height from the center y-coordinate to get the top boundary; and add half the height to the center y-coordinate to get the bottom boundary.

[0190] Specifically, when the left or top boundary is less than 0 pixels, it is adjusted to 0 pixels; when the right or bottom boundary is greater than 639 pixels, it is adjusted to 639 pixels.

[0191] After completing the boundary constraints, calculate the width and height of the character area proposal box, and delete character area proposal boxes with a width less than 8 pixels or a height less than 16 pixels.

[0192] In this embodiment, the remaining character region proposal boxes are arranged in descending order of character confidence, and non-maximum suppression is performed.

[0193] Specifically, first, the proposal box for the first person region is retained, and the intersection-union ratio (IUR) between this proposal box and the other proposal boxes for the other person regions is calculated. When the IUR is greater than 0.50, the proposal boxes with smaller person confidence values ​​are deleted. When the IUR is less than or equal to 0.50, the corresponding proposal boxes are retained. Then, the proposal box with the largest person confidence value is selected from the unprocessed proposal boxes, and the IUR calculation and deletion process is repeated until all proposal boxes are processed.

[0194] As a preferred embodiment, the non-maximum suppression threshold is determined based on the person recall rate and the proportion of duplicate proposal boxes in the verification image.

[0195] Specifically, validation was performed using thresholds of 0.30, 0.40, 0.50, and 0.60. Person recall was calculated by dividing the number of person bounding boxes correctly covered by retained proposals by the total number of person bounding boxes. The duplicate proposal ratio was calculated by dividing the number of duplicate proposals pointing to the same person bounding box by the total number of retained proposals. Subtracting the duplicate proposal ratio from the person recall yielded the screening evaluation value corresponding to each candidate threshold. The screening evaluation value corresponding to the threshold of 0.50 was the maximum; therefore, the non-maximum suppression threshold was set to 0.50.

[0196] Furthermore, after performing nonmaximum suppression, the top 100 character region proposal boxes are retained in descending order of character confidence, forming the character region proposal box set.

[0197] Each character region proposal box in the character region proposal box set records the coordinates of its left boundary, top boundary, right boundary, and bottom boundary, as well as the character confidence score.

[0198] Furthermore, local perceptual features are extracted from the initial feature map based on the set of proposed bounding boxes for the character region.

[0199] Specifically, the left, top, right, and bottom boundary coordinates of the character region proposal box in the scene image are first divided by 8 to obtain the corresponding boundary coordinates of the character region proposal box on the initial feature map. The coordinate conversion results are kept as decimals without rounding to preserve the character boundary position.

[0200] In this embodiment, the corresponding region of each character region proposal box on the initial feature map is divided into 7 rows and 7 columns, for a total of 49 pooling units; each pooling unit has the same size in the horizontal and vertical directions; each pooling unit is set with 4 sampling positions, which are located at the intersection of the horizontal quarter position, the horizontal three-quarter position and the vertical quarter position, and the vertical three-quarter position of the pooling unit, respectively.

[0201] For each sampling location, four feature locations around it are determined.

[0202] Specifically, the interpolation weights are determined based on the horizontal and vertical distances between the sampling location and the four feature locations. The feature location closest to the sampling location receives the largest interpolation weight, and the sum of the four interpolation weights is 1. The feature values ​​of the four feature locations are multiplied by their corresponding interpolation weights, and the four products are summed to obtain the feature value of the sampling location.

[0203] In this process, the feature values ​​of the four sampling positions in the same pooling unit are added together, and then the sum is divided by 4 to obtain the output feature value of the pooling unit in the current channel.

[0204] The above sampling and averaging processes were performed on each of the 256 channels to obtain the feature values ​​of the 256 channels corresponding to one pooling unit.

[0205] The same processing was performed sequentially on 49 pooling units to obtain local receptive features with 7 rows, 7 columns, and 256 channels.

[0206] Furthermore, a correspondence is established between each local perceptual feature and the person region proposal box that generated the feature, and the boundary coordinates of the person region proposal box and the person confidence are attached to the local perceptual feature, thereby obtaining a local perceptual feature containing the person bounding box.

[0207] S3 includes obtaining local receptive features containing the bounding box of the person and the confidence level corresponding to the local receptive features; if the confidence level is greater than the preset confidence level threshold, then the affine transformation matrix is ​​determined based on the coordinates of the bounding box of the person; the local receptive features are spatially mapped using the affine transformation matrix to obtain the mapped feature pixel distribution; the cropping operation area is determined in the mapped feature pixel distribution and the cropping operation is performed to obtain the aligned person identity features.

[0208] In this embodiment, the local receptive features, person bounding boxes, and person confidence scores output in step two are obtained. Each local receptive feature corresponds one-to-one with a person bounding box, and each local receptive feature includes 7 rows, 7 columns, and 256 feature channels.

[0209] Specifically, the bounding box of a person includes the coordinates of the left boundary, the upper boundary, the right boundary, and the lower boundary. The confidence score of a person is the classification probability output by the region proposal network for the bounding box of that person, and its value ranges from 0 to 1.

[0210] Furthermore, the preset confidence threshold is determined using a validation image independent of the training image after the neural network training is completed.

[0211] Specifically, 0.60, 0.65, 0.70, 0.75, and 0.80 were used as candidate thresholds in sequence, and the bounding boxes of people in the verification image were filtered according to each candidate threshold. For each candidate threshold, the number of bounding boxes of people with a confidence level greater than the candidate threshold and which correctly correspond to the bounding boxes of people was counted, and this number was divided by the total number of bounding boxes of people with a confidence level greater than the candidate threshold to obtain the corresponding filtering accuracy. The number of bounding boxes of people that were correctly covered by at least one bounding box of people was counted, and this number was divided by the total number of bounding boxes of people in the verification image to obtain the corresponding filtering recall.

[0212] The correct correspondence refers to the intersection-union ratio between the character's bounding box and the character's annotation box being greater than or equal to 0.50.

[0213] The calculation process for the intersection-union ratio is as follows: First, determine the overlapping area between the character's bounding box and the character's annotation box. Multiply the width of the overlapping area by the height of the overlapping area to obtain the overlapping area. Then, add the area of ​​the character's bounding box to the area of ​​the character's annotation box, and subtract the overlapping area from the sum to obtain the union area. Finally, divide the overlapping area by the union area to obtain the intersection-union ratio.

[0214] Furthermore, the screening precision and screening recall corresponding to the same candidate threshold are multiplied, the product is multiplied by 2, and then divided by the sum of screening precision and screening recall to obtain the evaluation value of the candidate threshold.

[0215] Among the validation results, the candidate threshold of 0.75 corresponds to the highest evaluation value, therefore the confidence threshold is fixed at 0.75. In subsequent image classification processes, this confidence threshold will no longer change with the input image.

[0216] Furthermore, the confidence level of the person corresponding to each local sensory feature is read.

[0217] Specifically, when the confidence level of the person is less than or equal to 0.75, the processing of the local perceptual feature is stopped; when the confidence level of the person is greater than 0.75, the local perceptual feature and the person's bounding box are retained, and the affine transformation matrix is ​​determined based on the coordinates of the person's bounding box.

[0218] In this embodiment, the coordinates of the top left corner are determined based on the coordinates of the left and top boundaries of the character's bounding box, the coordinates of the top right corner are determined based on the coordinates of the right and top boundaries, and the coordinates of the bottom left corner are determined based on the coordinates of the left and bottom boundaries.

[0219] Use the three corner points mentioned above as the original coordinate points, and set three target coordinate points corresponding to them.

[0220] In this embodiment, the target mapping area is set to 112 rows and 56 columns.

[0221] Specifically, the coordinates of the top left corner of the target mapping area are set to column 1 and row 1, the coordinates of the top right corner are set to column 56 and row 1, and the coordinates of the bottom left corner are set to column 1 and row 112. The coordinates of the top left corner of the character's bounding box are mapped to the coordinates of the top left corner of the target mapping area, the coordinates of the top right corner of the character's bounding box are mapped to the coordinates of the top right corner of the target mapping area, and the coordinates of the bottom left corner of the character's bounding box are mapped to the coordinates of the bottom left corner of the target mapping area.

[0222] Specifically, the affine transformation matrix includes two sets of coordinate transformation parameters, each set including two linear transformation parameters and one translation parameter.

[0223] Specifically, the three original coordinate points of the character's bounding box are substituted into the horizontal coordinate transformation relationship to form three linear constraints on the horizontal transformation parameters; the three original coordinate points are substituted into the vertical coordinate transformation relationship to form three linear constraints on the vertical transformation parameters; the above two sets of linear constraints are solved to obtain six transformation parameters, which constitute an affine transformation matrix.

[0224] In addition, before performing coordinate calculations, the coordinates of the person's bounding box in the scene image are converted into coordinates in the local receptive features.

[0225] Specifically, the coordinates of the left, right, top, and bottom boundaries of the character bounding box in the scene image are divided by 8 to obtain its corresponding coordinates in the initial feature map. Then, using the left and top boundaries of the character bounding box in the initial feature map as the coordinate starting points of the local receptive features, the horizontal coordinate range of the character bounding box is converted to the range of 7 columns of the local receptive features, and the vertical coordinate range of the character bounding box is converted to the range of 7 rows of the local receptive features, so that the corner points of the character bounding box and the local receptive features use the same coordinate reference.

[0226] Furthermore, when mapping local receptive features to spatial coordinates using an affine transformation matrix, a reverse coordinate lookup method is employed.

[0227] Specifically, for each target location in the target mapping region, the source coordinates of the target location in the local receptive features are determined using the inverse transformation result of the affine transformation matrix; the source coordinates include row coordinates and column coordinates, and the calculation results are rounded to decimals.

[0228] When the source coordinates coincide with the integer coordinates in the local receptive features, the feature values ​​of 256 channels at that integer coordinate are directly read.

[0229] When the source coordinates are located between four adjacent feature locations, bilinear interpolation is used to calculate the feature value of the source coordinates.

[0230] The specific process of bilinear interpolation in this embodiment is as follows: determine the four adjacent feature positions of the source coordinates at the upper left, upper right, lower left, and lower right; calculate the horizontal and vertical distances between the source coordinates and the four adjacent feature positions respectively; and determine the interpolation weights of the four adjacent feature positions based on the horizontal and vertical distances.

[0231] When the distance between the source coordinates and the adjacent feature positions decreases, the interpolation weight of the adjacent feature position increases, and the sum of the four interpolation weights is 1.

[0232] Specifically, for the same feature channel, the feature values ​​of the four adjacent feature positions are multiplied by their corresponding interpolation weights, and then the four products are added together. The result is used as the mapping value of the target position on that feature channel.

[0233] The above calculations are performed sequentially on the 256 feature channels to obtain the feature values ​​of the 256 channels corresponding to the target position; the positions of row 112 and column 56 in the target mapping area are processed in the same way to form the mapped feature pixel distribution of row 112, column 56 and 256 feature channels.

[0234] When the source coordinates exceed the range of 7 rows and 7 columns of the local sensing feature, the feature values ​​of all 256 channels at the corresponding target location are set to 0 to prevent reading data outside the boundary of the local sensing feature.

[0235] Furthermore, after completing the spatial coordinate mapping, the cropping operation area is determined in the mapped feature pixel distribution.

[0236] In this embodiment, the size and position of the cropping operation area are predetermined using a verification image.

[0237] Specifically, the center of the target mapping region with 112 rows and 56 columns is used as the clipping center, and four candidate clipping regions with 96 rows and 48 columns, 96 rows and 44 columns, 104 rows and 48 columns, and 104 rows and 44 columns are set respectively.

[0238] Furthermore, four candidate cropping regions are used to extract the mapped feature pixel distribution in the verification image, and the cropping results are input into the same person identity classification layer.

[0239] Specifically, the number of correctly classified images corresponding to each candidate cropping region is counted, and this number is divided by the total number of verification images to obtain the identity classification accuracy. In the verification results, the identity classification accuracy corresponding to the candidate cropping region with 96 rows and 48 columns is the highest. Therefore, the cropping operation region is fixed at 96 rows and 48 columns.

[0240] The target mapping area has a width of 56 columns, and the cropping operation area has a width of 48 columns, which is 8 columns apart. Therefore, 4 columns are deleted from the left side of the target mapping area and 4 columns are deleted from the right side.

[0241] The target mapping area has a height of 112 rows, and the clipping operation area has a height of 96 rows, a difference of 16 rows. Therefore, 8 rows are deleted from the top of the target mapping area, and 8 rows are deleted from the bottom.

[0242] Accordingly, rows 9 to 104 of the mapped feature pixel distribution are defined as the vertical range of the cropping operation area, and columns 5 to 52 are defined as the horizontal range of the cropping operation area; data from all 256 feature channels within the above range are read to obtain cropping features of 96 rows, 48 ​​columns, and 256 feature channels.

[0243] As a preferred embodiment, the clipping feature is adjusted to 28 rows and 14 columns, while the number of channels remains unchanged at 256.

[0244] Specifically, the size adjustment uses bilinear interpolation. For each adjusted position, based on its corresponding coordinates in the 96 rows and 48 columns of the clipping feature, the feature values ​​of the four surrounding adjacent positions are read, and the output values ​​of 256 channels are calculated according to the aforementioned bilinear interpolation process.

[0245] After resizing, the aligned character identity features are obtained with 28 rows, 14 columns, and 256 feature channels.

[0246] S4 includes obtaining aligned character identity features, extracting the identity responses of the character identity features to obtain a response distribution matrix; calculating the numerical difference based on the response distribution matrix to obtain a set of decay gradients of the identity responses; determining the region segmentation result based on the set of decay gradients; using the region segmentation result to determine the character region and assign boundary weights to obtain semantic hierarchical features with boundary weights; distinguishing hierarchical differences based on semantic hierarchical features to obtain a mask weight map used to distinguish semantic hierarchical differences.

[0247] In this embodiment, the aligned identity features of the person are obtained from the output.

[0248] Specifically, the identity features of the person include 28 rows, 14 columns and 256 feature channels. Each spatial location corresponds to a local location in the person's area, and the feature response of that local location on the 256 channels is recorded.

[0249] In the process of extracting identity responses from the characteristics of a person, a convolution kernel with a size of 1 feature unit by 1 feature unit is used to compress the 256 channel features of each spatial location into 1 identity response value.

[0250] Specifically, the 256 channel feature values ​​of the spatial location are multiplied by the corresponding 256 channel weights in the convolution kernel, the resulting products are added together, and then the bias value of the convolution kernel is superimposed to obtain the original identity response of the spatial location.

[0251] Furthermore, the channel weights and biases of the convolution kernel are determined during the training phase.

[0252] Specifically, spatial locations within the character annotation area in the training image are assigned a value of 1, while spatial locations outside the character annotation area are assigned a value of 0. The position-by-position classification loss is calculated based on the difference between the original identity response obtained from convolution processing and the corresponding assigned value, and the channel weights and bias values ​​are updated based on the classification loss. After training is completed, the channel weights and bias values ​​are fixed and used for identity response extraction of the image to be classified.

[0253] Furthermore, the values ​​of each original identity response are normalized.

[0254] During normalization, the negative number of the original identity response is taken first, and then the natural exponent value of the negative number is calculated. The natural exponent value is added to 1, and 1 is divided by the sum to obtain the identity response value between 0 and 1. All identity response values ​​are arranged according to the original row and column positions of the person's identity characteristics to obtain a response distribution matrix with 28 rows and 14 columns.

[0255] Furthermore, the values ​​in the response distribution matrix are used to characterize the degree of correlation between the corresponding location and the person's identity information.

[0256] Specifically, when the identity response value is close to 1, the location belongs to the character area; when the identity response value is close to 0, the location belongs to the background area; when the identity response value is between 0 and 1, the location is in the transition area between the character outline and the background.

[0257] Furthermore, after obtaining the response distribution matrix, the numerical difference between adjacent responses is calculated position by position.

[0258] Specifically, for any spatial location, the identity response values ​​of its left and right adjacent locations are read. The left identity response value is subtracted from the right identity response value. The absolute value of the difference is taken and then divided by 2 to obtain the lateral response change of that spatial location.

[0259] In this embodiment, the identity response values ​​of the upper and lower adjacent positions of the spatial location are read, the identity response value of the upper position is subtracted from the identity response value of the lower position, the absolute value of the difference is taken, and then the absolute value is divided by 2 to obtain the longitudinal response change of the spatial location.

[0260] Specifically, for the spatial position in column 1, the non-existent left identity response value is replaced by the identity response value of that position itself; for the spatial position in column 14, the non-existent right identity response value is replaced by the identity response value of that position itself; for the spatial position in row 1, the non-existent upper identity response value is replaced by the identity response value of that position itself; and for the spatial position in row 28, the non-existent lower identity response value is replaced by the identity response value of that position itself.

[0261] Furthermore, the change in lateral response is multiplied by itself, the change in longitudinal response is multiplied by itself, the two multiplication results are added together, and the square root of the sum is calculated to obtain the attenuation gradient value at that spatial location.

[0262] The above calculations are performed on each of the 392 spatial locations in the response distribution matrix, and the resulting attenuation gradient values ​​are saved according to their original row and column positions to form a set of attenuation gradients for the identity response.

[0263] As a preferred embodiment, the identity response threshold and decay gradient threshold used for region segmentation are determined by verifying images after the neural network training is completed.

[0264] During verification, 0.55, 0.60, and 0.65 were used as candidate response thresholds for the human region, 0.25, 0.30, and 0.35 were used as candidate response thresholds for the background region, and 0.08, 0.10, and 0.12 were used as candidate attenuation gradient thresholds. The candidate thresholds were combined one by one to generate the corresponding region segmentation results and mask weight maps. Then, the scene environment features were extracted using each mask weight map to complete the scene classification.

[0265] For each set of candidate thresholds, the number of images whose classification results match the true category of the verification images is counted, and this number is divided by the total number of verification images to obtain the classification accuracy corresponding to that set of candidate thresholds.

[0266] The candidate threshold combination with the highest classification accuracy is determined as the region partitioning parameter.

[0267] In this embodiment, the response threshold for the human region is determined to be 0.60, the response threshold for the background region is determined to be 0.30, and the attenuation gradient threshold is determined to be 0.10; the above thresholds remain unchanged during the image classification stage.

[0268] For each spatial location, read the identity response value and attenuation gradient value sequentially; when the identity response value is greater than or equal to 0.60 and the attenuation gradient value is less than or equal to 0.10, divide the spatial location into the character's internal region.

[0269] When the identity response value is less than or equal to 0.30, the spatial location is divided into the background region; the background region retains its original feature response in the subsequent scene environment feature extraction process.

[0270] When the identity response value is greater than 0.30 and less than 0.60, the spatial location is divided into the character boundary region; when the identity response value is greater than or equal to 0.60, but the decay gradient value is greater than 0.10, it indicates that the location is at a position where the identity response changes abruptly, and it is also divided into the character boundary region.

[0271] In addition, to distinguish the semantic levels in the character boundary region, the character boundary region is divided into 3 layers according to the identity response value; when the identity response value is greater than or equal to 0.50 and less than 0.60, the corresponding position is determined as the first boundary layer; when the identity response value is greater than or equal to 0.40 and less than 0.50, the corresponding position is determined as the second boundary layer; when the identity response value is greater than 0.30 and less than 0.40, the corresponding position is determined as the third boundary layer; the position with an identity response value greater than or equal to 0.60 and a decay gradient value greater than 0.10 is assigned to the first boundary layer.

[0272] Furthermore, after completing the regional division, boundary weights are assigned to each region.

[0273] Specifically, the weight of the area inside the character is set to 0, the weight of the first boundary layer is set to 0.25, the weight of the second boundary layer is set to 0.50, the weight of the third boundary layer is set to 0.75, and the weight of the background area is set to 1.

[0274] In this embodiment, corresponding weights are written according to the row and column positions of the response distribution matrix to form a semantic hierarchical feature with 28 rows and 14 columns.

[0275] The position with a value of 0 corresponds to the internal area of ​​the character, the positions with values ​​of 0.25, 0.50 and 0.75 correspond to the three boundary layers respectively, and the position with a value of 1 corresponds to the background area; the semantic hierarchy differences between the internal area of ​​the character, the transition position of the character outline and the background position are recorded by different weights.

[0276] In addition, to eliminate anomalous responses at individual spatial locations, local smoothing is applied to semantic hierarchical features.

[0277] Specifically, taking each spatial location as the center, the weight values ​​within a range of 3 rows and 3 columns around it are read. All the weight values ​​actually read are added together, and the sum is divided by the number of spatial locations involved in the calculation. The average value is used as the smoothed weight of that spatial location.

[0278] Furthermore, the spatial location at the edge of the semantic level feature is read only from the actual adjacent locations within the matrix range.

[0279] Specifically, when the smoothed weight is less than 0, it is adjusted to 0; when the smoothed weight is greater than 1, it is adjusted to 1; when the smoothed weight is between 0 and 1, the weight is retained.

[0280] Furthermore, after smoothing all spatial locations, a mask weight map is obtained to distinguish semantic level differences.

[0281] Specifically, the mask weight map contains 28 rows and 14 columns. The weight of the area inside the character is 0, the weight of the background area is 1, and the weight of the area at the boundary of the character is between 0 and 1.

[0282] S5 includes obtaining a mask weight map and an initial feature map containing spatial distribution information; processing the mask weight map through feature dimension transformation to obtain an alignment weight matrix; extracting pixel values ​​of the person region based on the alignment weight matrix to generate an inverse mask matrix; if the element value of the inverse mask matrix is ​​greater than a preset threshold, determining that the region corresponding to the element value is the background pixel retention region to obtain a background mask map; performing matrix multiplication between the background mask map and the initial feature map to remove the person region in the initial feature map and obtain scene environment features.

[0283] In this embodiment, the generated mask weight map and the generated initial feature map are obtained.

[0284] Specifically, the mask weight map contains 28 rows and 14 columns, and the initial feature map contains 80 rows, 80 columns, and 256 feature channels. The element values ​​in the mask weight map are between 0 and 1, where the element value corresponding to the inner region of the person is 0, the element value corresponding to the boundary region of the person is a continuous value between 0 and 1, and the element value corresponding to the background region is 1.

[0285] Furthermore, the mask weight map corresponds to the cropped and aligned character identity region, and its spatial coordinates are inconsistent with the character bounding box coordinates in the initial feature map.

[0286] To establish a positional correspondence between the mask weight map and the initial feature map, the reserved coordinates of the character's bounding box are first read. Then, the coordinates of the character's bounding box in the scene image (left, top, right, and bottom boundaries) are each divided by 8 to obtain the coordinate range of the character's bounding box in the initial feature map. The coordinate conversion results are rounded to decimals.

[0287] Using the left and top boundaries of the character bounding box in the initial feature map as the mapping start point and the right and bottom boundaries as the mapping end point, the 14 columns of the mask weight map are mapped to the horizontal range of the character bounding box, and the 28 rows of the mask weight map are mapped to the vertical range of the character bounding box.

[0288] Specifically, for any target location within the character's bounding box, first subtract the left boundary coordinate of the character's bounding box from the column coordinate of the target location, then divide the difference by the difference between the right and left boundary coordinates of the character's bounding box to obtain the position ratio of the target location within the horizontal range of the character's bounding box; multiply this position ratio by 13 to obtain its horizontal source coordinates in the mask weight map.

[0289] Using the same method, subtract the upper boundary coordinates of the character's bounding box from the row coordinates of the target position, divide the difference by the difference between the lower and upper boundary coordinates of the character's bounding box, and obtain the position ratio of the target position within the vertical range of the character's bounding box; multiply this position ratio by 27 to obtain its vertical source coordinates in the mask weight map.

[0290] When both the horizontal and vertical source coordinates are integers, the weight values ​​at the corresponding positions in the mask weight map are read directly.

[0291] When the source coordinates contain decimals, determine the four positions around the source coordinates: top left, top right, bottom left, and bottom right, and use bilinear interpolation to calculate the weight value of the target position.

[0292] In addition, during bilinear interpolation, the horizontal and vertical distances are first determined based on the decimal part of the source coordinates.

[0293] Specifically, the horizontal weight at the left position is equal to 1 minus the horizontal decimal part, and the horizontal weight at the right position is equal to the horizontal decimal part; the vertical weight at the top position is equal to 1 minus the vertical decimal part, and the vertical weight at the bottom position is equal to the vertical decimal part; the horizontal weights and vertical weights are multiplied respectively to obtain the interpolation weights corresponding to the four adjacent positions; the sum of the four interpolation weights is 1; the mask weights of the four adjacent positions are multiplied by the corresponding interpolation weights respectively, and the four products are added together, and the result is used as the weight value of the target position.

[0294] Specifically, for spatial locations outside the character's bounding box, the weight value is set to 1.

[0295] By traversing the 80 rows and 80 columns of the initial feature map using the above processing method, a spatial weight matrix with 80 rows and 80 columns is obtained.

[0296] In this embodiment, when there are two bounding boxes for two people in the scene image, the two mask weight maps are mapped to the initial feature map respectively to obtain two spatial weight matrices.

[0297] Specifically, for the same spatial location, two weight values ​​are compared, and the weight with the smaller value is written to that location; when there are more than three character bounding boxes in the scene image, the same position-by-position comparison is performed on all spatial weight matrices, and the smallest weight is retained; thus, the position corresponding to any character's internal region retains a suppression weight with a value of 0.

[0298] Furthermore, after completing the spatial mapping, the 80-row, 80-column spatial weight matrix is ​​copied 256 times along the channel direction so that each channel has the same spatial weight, resulting in an alignment weight matrix with 80 rows, 80 columns, and 256 channels.

[0299] Furthermore, when extracting pixel values ​​of the character area based on the alignment weight matrix, the weight of each spatial position is read one by one.

[0300] The threshold for extracting human figures is fixed at 0.50. When the weight value of a spatial location is less than or equal to 0.50, the location is identified as a human figure region and the human figure region marker is set to 1; when the weight value is greater than 0.50, the location is identified as a non-human figure region and the human figure region marker is set to 0.

[0301] As a preferred embodiment, the person region extraction threshold is determined by verifying the image.

[0302] Specifically, in the verification stage, 0.40, 0.45, 0.50, 0.55 and 0.60 are used as candidate thresholds in sequence, and corresponding human figures regions are generated using each candidate threshold. Then, the generated human figures regions are compared with the human figures labeled regions in the verification image.

[0303] During the comparison, the number of spatial locations that overlap between the person region and the person annotation region is counted, multiplied by 2, and then the result is divided by the sum of the number of spatial locations contained in the person region and the number of spatial locations contained in the person annotation region to obtain the region overlap evaluation value. The region overlap evaluation values ​​of all verification images are summed and then divided by the number of verification images to obtain the average evaluation value corresponding to the candidate threshold. The average evaluation value corresponding to the candidate threshold of 0.50 is the largest, so the threshold for extracting the person region is determined to be 0.50.

[0304] Furthermore, after obtaining the character region marking results, the character region marking value at each spatial location is subtracted from 1 to form an inverse mask matrix.

[0305] In this process, the positions marked as 1 in the character area are reversed to become 0, and the positions marked as 0 in the non-character area are reversed to become 1; thus, the reverse mask matrix forms a spatial distribution with the character area as 0 and the positions outside the character area as 1.

[0306] To eliminate isolated elements caused by coordinate interpolation, a local averaging process of 3 rows by 3 columns is performed once on the inverse mask matrix.

[0307] For any spatial location, read the element values ​​within a 3x3 range centered on that location, add the actual element values ​​read, divide the sum by the number of elements involved in the calculation, and use the average value as the processed element value for that spatial location; for matrix edge locations, only the elements that actually exist within the matrix range are calculated.

[0308] Furthermore, after local averaging, the element values ​​in the inverse mask matrix are between 0 and 1. Each element value is compared with a preset threshold to determine the background pixel retention area.

[0309] Specifically, the preset threshold is determined by the verification image; in the verification stage, 0.40, 0.50, 0.60 and 0.70 are used as candidate thresholds in sequence; for each candidate threshold, the position in the inverse mask matrix that is greater than the candidate threshold is determined as the background pixel retention area, and a background mask image is formed accordingly; then the scene environment features are extracted using the background mask image, and input into the subsequent classification network to complete the classification of the verification image.

[0310] In this embodiment, the number of verification images whose classification results match the real scene category is counted, and this number is divided by the total number of verification images to obtain the classification accuracy of the corresponding candidate threshold.

[0311] Among them, the classification accuracy is highest when the candidate threshold is 0.50. Therefore, the preset threshold is set to 0.50 and remains unchanged during the image processing.

[0312] Specifically, when the value of an element in the inverse mask matrix is ​​greater than 0.50, the position corresponding to that element is determined as the background pixel retention area, and 1 is written at the corresponding position in the background mask image.

[0313] When the value of an element in the inverse mask matrix is ​​less than or equal to 0.50, the position is determined to be a character area, and 0 is written in the corresponding position of the background mask. Traverse all positions in the inverse mask matrix to obtain a background mask with 80 rows and 80 columns.

[0314] Furthermore, the background mask image is copied 256 times along the channel direction so that the background mask image has the same spatial size and number of channels as the initial feature image.

[0315] Then, according to the same row position, the same column position, and the same channel position, the element values ​​in the background mask image are multiplied one by one with the feature values ​​in the initial feature image.

[0316] In this embodiment, the position with an element value of 0 in the background mask image corresponds to the person area; after multiplying this element value by the 256 channel feature values ​​at the corresponding position in the initial feature image, all channel feature values ​​become 0, thereby eliminating the feature response of the person area.

[0317] Furthermore, the positions in the background mask image where the element value is 1 correspond to the background pixel retention area.

[0318] Specifically, after multiplying the element value with the corresponding 256 channel feature values ​​in the initial feature map, the feature values ​​of each channel remain unchanged, thus preserving the spatial structure and semantic response of the scene background; after completing the position-by-position multiplication of 80 rows, 80 columns and 256 channels, the scene environment features are obtained.

[0319] S6 includes acquiring scene environment features and character identity features, performing dimensional alignment processing on the scene environment features and character identity features to obtain aligned features; performing semantic mapping processing on the aligned features to obtain an associated feature matrix; performing interference filtering processing on the associated feature matrix to determine a pure feature vector, wherein the pure feature vector is generated by adaptive pooling operation; and performing feature concatenation and global aggregation processing on the pure feature vector to obtain a fused feature expression that eliminates competition interference.

[0320] In this embodiment, the generated scene environment features and the generated aligned character identity features are obtained.

[0321] Specifically, the scene environment features include 80 rows, 80 columns, and 256 feature channels, where the feature values ​​of the corresponding positions in the character area have been set to 0; the character identity features include 28 rows, 14 columns, and 256 feature channels, used to record the identity response formed after spatial alignment and cropping of the character area.

[0322] Since the two features have different spatial dimensions, adaptive average pooling is first applied to the scene environment features to adjust their spatial dimensions from 80 rows and 80 columns to 28 rows and 14 columns, while keeping the number of channels unchanged at 256.

[0323] In this embodiment, when performing adaptive average pooling, the 80 rows of scene environment features are divided into 28 consecutive row intervals, and the 80 columns are divided into 14 consecutive column intervals.

[0324] Specifically, for the first output row position, the value obtained by dividing 80 by 28 determines the ending position of the corresponding input row range; with 0 as the starting position, the starting position is rounded down and the ending position is rounded up to obtain the input row participating in the calculation; the remaining output row positions are determined in the same way; the output column position is determined by dividing 80 by 14 to determine the corresponding input column range.

[0325] Furthermore, for any output position, all feature values ​​of the same channel within its corresponding row and column intervals are read, the read feature values ​​are added together, and the sum is divided by the number of spatial positions involved in the calculation. The average value is then written to that output position.

[0326] After performing the above processing on each of the 256 channels, we obtained environment size alignment features with 28 rows, 14 columns, and 256 channels.

[0327] Subsequently, channel mapping was performed on the environmental size alignment features and the person identity features respectively.

[0328] Specifically, channel mapping uses convolutional layers with a size of 1 feature unit x 1 feature unit, and the network parameters are configured for each of the two convolutional layers. For any spatial location, the 256 channel feature values ​​at that location are multiplied by the corresponding 256 channel weights in the convolutional layer, the 256 products are added together, and then the bias value of the corresponding output channel is superimposed to obtain a mapped feature value. Each convolutional layer has 256 output channels, so the environmental features and the person's identity features after channel mapping both maintain 28 rows, 14 columns, and 256 channels.

[0329] Furthermore, the two features after channel mapping are subjected to batch normalization and linear rectification, respectively.

[0330] Specifically, batch normalization adjusts the feature distribution of a channel based on the mean and variance of the same channel in the current training batch; linear rectification sets feature values ​​less than 0 to 0 and retains feature values ​​greater than or equal to 0; after the above processing is completed, aligned features with consistent spatial size and number of channels are obtained.

[0331] The 28 rows and 14 columns together form 392 spatial locations.

[0332] For each spatial location in the environment alignment feature, one environment feature vector is extracted according to the arrangement of 256 channels; for each spatial location in the person identity feature, one person feature vector is extracted according to the same channel order; thus, 392 environment feature vectors and 392 person feature vectors are obtained respectively, and each feature vector contains 256 values.

[0333] Furthermore, semantic mapping is used to calculate the degree of feature association between environmental spatial location and person spatial location.

[0334] For any environmental feature vector, multiply each of its 256 values ​​by itself, sum the 256 products, and then perform a square root operation on the sum to obtain the length of the environmental feature vector; calculate the length of the human feature vector using the same process.

[0335] Specifically, the values ​​of the environmental feature vector and the human feature vector at the same channel position are multiplied respectively, and then the 256 products are added together to obtain the channel product sum. The channel product sum is divided by the product of the length of the environmental feature vector and the length of the human feature vector to obtain the semantic association value between the two spatial positions. When the length of the environmental feature vector or the length of the human feature vector is 0, the corresponding semantic association value is set to 0 to avoid division by zero.

[0336] Furthermore, the semantic association values ​​between 392 environmental spatial locations and 392 personal spatial locations are calculated sequentially to obtain an association feature matrix with 392 rows and 392 columns.

[0337] In this system, each row of the association feature matrix corresponds to one environmental spatial location, each column corresponds to one person spatial location, and each element is used to record the degree of semantic association between the corresponding environmental location and the person's location.

[0338] In addition, the semantic association value ranges from -1 to 1.

[0339] To unify the numerical range of subsequent interference filtering, each semantic association value is incremented by 1, and the result is divided by 2, so that the element values ​​in the association feature matrix are converted to between 0 and 1.

[0340] Furthermore, for each row in the correlation feature matrix, the 392 element values ​​in that row are compared, and the one with the largest value is determined as the disturbance response value for that environmental spatial location.

[0341] Specifically, the interference response value is used to characterize the semantics of a person's identity remaining in the environmental spatial location; when the environmental spatial location still retains the outline, clothing texture or limb structure response of a person, the location and the corresponding location in the person's identity features form an interference response with a value greater than 0.60.

[0342] Furthermore, the interference filtering threshold is determined using validation images after the network training is completed.

[0343] During verification, candidate thresholds of 0.50, 0.55, 0.60, 0.65, and 0.70 were used sequentially. For each candidate threshold, environmental locations with interference response values ​​greater than that threshold were masked, and scene classification was completed using the filtered environmental features and human identity features. The number of verification images whose classification results matched the true category was counted, and this number was divided by the total number of verification images to obtain the classification accuracy for the corresponding candidate threshold. The candidate threshold of 0.60 had the highest classification accuracy, so the interference filtering threshold was set to 0.60 and remained unchanged during the classification stage.

[0344] When the interference response value at a certain location in the environment is greater than 0.60, the filtering weight at that location is set to 0.

[0345] When the interference response value is less than or equal to 0.60, subtract the interference response value from 1 and determine the resulting value as the filtering weight for that location; multiply the feature values ​​of the environmental spatial location on each of the 256 channels by the filtering weight to obtain the filtered environmental features.

[0346] For example, when the interference response value of a certain environmental location is 0.25, its filtering weight is 0.75, and all 256 channel feature values ​​of that location are retained at a ratio of 0.75; when the interference response value is 0.60, the filtering weight is 0.40; when the interference response value is greater than 0.60, the filtering weight is 0, and all 256 channel feature values ​​of that location are set to 0.

[0347] When the disturbance response values ​​of all 392 environmental spatial locations are greater than 0.60, compare all disturbance response values, retain the environmental spatial location with the smallest value, and set its filtering weight to 1. Set the filtering weight of the remaining environmental spatial locations to 0 to ensure that the subsequent pooling process has non-zero input.

[0348] Furthermore, adaptive average pooling with an output size of 1 row and 1 column is performed on the filtered environmental features.

[0349] For any channel, read the environmental spatial locations with a filter weight greater than 0, add the feature values ​​corresponding to the above locations, and then divide the sum by the number of environmental spatial locations involved in the calculation to obtain the environmental feature value of that channel; process the 256 channels in sequence to obtain a clean scene environmental feature vector containing 256 values.

[0350] Perform adaptive average pooling on the character's identity features, with an output size of 1 row and 1 column.

[0351] For any given channel, the feature values ​​of that channel at 392 spatial locations are summed, and the sum is divided by 392 to obtain the corresponding character identity feature value. After pooling 256 channels, a clean character identity feature vector containing 256 values ​​is obtained.

[0352] In this embodiment, the pure human identity feature vector is arranged first, and the pure scene environment feature vector is arranged second. They are then connected end to end according to the channel order to form a spliced ​​feature containing 512 values.

[0353] The first 256 values ​​correspond to the person's identity information, and the last 256 values ​​correspond to the scene environment information after interference filtering.

[0354] Furthermore, the spliced ​​features are input into the global aggregation layer.

[0355] Specifically, the global aggregation layer is configured with 512 input nodes and 512 output nodes. For any output node, the 512 values ​​in the concatenated feature are multiplied by the 512 connection weights corresponding to that output node, the products are summed, and then the bias value of that output node is added to obtain one aggregated feature value. The calculation is performed sequentially for the 512 output nodes to obtain a global aggregated feature containing 512 values.

[0356] After batch normalization and linear rectification, the global aggregated features are processed to form a fused feature expression that eliminates competition interference.

[0357] S7 includes acquiring a fusion feature representation that eliminates competition interference, extracting character identity features and scene environment features from the fusion feature representation and calculating the initial activation intensity; calculating the co-discrimination gain based on the initial activation intensity; if the co-discrimination gain is greater than a preset threshold, the first channel feature is obtained; if the co-discrimination gain is less than or equal to the preset threshold, the second channel feature is obtained; and adaptively compressing the first channel feature and the second channel feature based on the co-discrimination gain to obtain the dimensionality-reduced discriminative features.

[0358] In this embodiment, the fusion feature representation of the output is obtained.

[0359] Specifically, the fused feature representation contains 512 feature values, which already include person identity information and scene environment information, and have completed competitive interference filtering and global aggregation processing.

[0360] To extract character identity features and scene environment features separately, the fused feature representation is simultaneously input into the character identity mapping branch and the scene environment mapping branch. Both the character identity mapping branch and the scene environment mapping branch are configured with 512 input nodes and 256 output nodes, and the two mapping branches are configured with connection weights and bias parameters respectively.

[0361] Specifically, for any output node in the character identity mapping branch, the 512 feature values ​​in the fused feature expression are multiplied by the 512 connection weights corresponding to the output node, and the resulting 512 products are added together. The bias value corresponding to the output node is then added to the sum to obtain one character identity mapping value. The same process is used to calculate 256 output nodes to obtain character identity features containing 256 character identity mapping values.

[0362] In this embodiment, the scene environment mapping branch uses the same node calculation method.

[0363] Specifically, for each output node, the 512 feature values ​​in the fused feature representation are multiplied one by one with the 512 connection weights corresponding to that output node. The resulting products are summed and then superimposed with the bias value to obtain one scene environment mapping value. After completing the calculation for 256 output nodes, a scene environment feature containing 256 scene environment mapping values ​​is obtained.

[0364] Furthermore, the parameters of the two mapping branches are determined during the neural network training phase.

[0365] Specifically, the character identity mapping branch adjusts the connection weights and bias parameters based on the classification loss between character identity features and scene category labels; the scene environment mapping branch adjusts the connection weights and bias parameters based on the classification loss between scene environment features and scene category labels; after training is completed, all parameters of the two mapping branches are fixed.

[0366] Furthermore, the character's identity features and scene environment features are activated separately to obtain the initial activation intensity corresponding to each channel.

[0367] For any character identity mapping value, first take the opposite of the mapping value, then calculate the natural exponent value of the opposite number; add the obtained natural exponent value to 1, and divide 1 by the above addition result to obtain the initial activation strength of the character identity with a value between 0 and 1.

[0368] The scene environment mapping values ​​of the corresponding channels are processed in the same way to obtain the initial activation intensity of the scene environment.

[0369] Thus, the initial activation strength of the first character identity corresponds to the initial activation strength of the first scene environment in the first channel, the initial activation strength of the second character identity corresponds to the initial activation strength of the second scene environment in the second channel, and so on, until 256 channels are formed.

[0370] For each channel, the collaborative discrimination gain is calculated based on the initial activation intensity of the character's identity and the initial activation intensity of the scene environment.

[0371] Specifically, the initial activation intensity of the character's identity in the same channel is multiplied by the initial activation intensity of the scene environment, and then the square root operation is performed on the multiplication result. The resulting value is used as the collaborative discrimination gain of that channel.

[0372] In addition, the value of the co-discrimination gain is between 0 and 1.

[0373] Specifically, when both the initial activation strength of the character's identity and the initial activation strength of the scene environment are 1, the collaborative discrimination gain of this channel is 1; when either initial activation strength is 0, the collaborative discrimination gain of this channel is 0. Through this calculation process, this channel can only form the corresponding collaborative discrimination gain when the character's identity response and the scene environment response exist simultaneously in the same channel.

[0374] Furthermore, the co-discriminative gain threshold is determined using a validation image independent of the training image after the neural network training is completed.

[0375] Specifically, during the verification phase, 0.45, 0.50, 0.55, 0.60, and 0.65 were used as candidate thresholds in sequence. For each candidate threshold, the first channel features and the second channel features were divided according to the candidate threshold, and the same compression layer and classification layer were used to complete the verification image classification.

[0376] Furthermore, the number of verification images whose classification results match the true category at each candidate threshold is counted, and then this number is divided by the total number of verification images to obtain the corresponding classification accuracy.

[0377] In this embodiment, the classification accuracy value corresponding to the candidate threshold of 0.55 is the largest, so the collaborative discrimination gain threshold is determined to be 0.55; this threshold remains unchanged during the processing of the image to be classified.

[0378] The collaborative discrimination gain of 256 channels was read sequentially and compared with 0.55.

[0379] Specifically, when the collaborative discrimination gain of a channel is greater than 0.55, the channel is assigned to the first channel set; when the collaborative discrimination gain of a channel is less than or equal to 0.55, the channel is assigned to the second channel set.

[0380] As a preferred embodiment, the first channel features are represented by a feature matrix with 2 rows and 256 columns.

[0381] Specifically, the first row is used to record the character identity feature value, and the second row is used to record the scene environment feature value. For channels with a collaborative discrimination gain greater than 0.55, the character identity feature value and scene environment feature value corresponding to that channel are written into the first row and the second row of the first channel feature, respectively. For channels with a collaborative discrimination gain less than or equal to 0.55, the two feature values ​​at the corresponding positions in the first channel feature are both set to 0.

[0382] Furthermore, the second channel features are also represented by a feature matrix with 2 rows and 256 columns.

[0383] Specifically, for channels with a collaborative discrimination gain less than or equal to 0.55, the corresponding person identity feature value and scene environment feature value are written into the first and second rows of the second channel feature, respectively; for channels with a collaborative discrimination gain greater than 0.55, the two feature values ​​at the corresponding positions in the second channel feature are both set to 0.

[0384] In addition, before compressing the features of the first channel, the character identity feature value and scene environment feature value are adjusted by using the collaborative discriminative gain of the corresponding channel.

[0385] For any first channel, the character identity feature value of the channel is multiplied by the corresponding collaborative discrimination gain, and the scene environment feature value is multiplied by the same collaborative discrimination gain to obtain two weighted feature values ​​for the channel.

[0386] For example, when the collaborative discrimination gain of a certain first channel is 0.80, the character identity feature value and scene environment feature value of that channel are multiplied by 0.80 respectively.

[0387] Therefore, the collaborative discrimination gain directly determines the characteristic amplitude of the channel when it enters the compression layer.

[0388] Furthermore, the first channel features are expanded in the order of row 1 first and row 2 last to obtain the first compressed input features containing 512 values.

[0389] The first compressed input feature is input into the first dimensionality reduction layer; the first dimensionality reduction layer includes 512 input nodes and 128 output nodes.

[0390] Specifically, for any output node in the first dimensionality reduction layer, the 512 values ​​in the first compressed input feature are multiplied by the 512 connection weights corresponding to the output node, the resulting products are added together, and then the bias value of the output node is added to obtain one first compressed feature value; after completing the calculation of 128 output nodes, the first compressed feature containing 128 feature values ​​is obtained.

[0391] Furthermore, the second channel features employ a proportional adjustment method based on collaborative discriminative gain.

[0392] For any second channel, the co-discrimination gain of that channel is divided by 0.55, and the resulting value is used as the compression retention ratio of that channel. Since the co-discrimination gain of the second channel is less than or equal to 0.55, the compression retention ratio is between 0 and 1.

[0393] Furthermore, the character identity feature value in the second channel is multiplied by the corresponding compression and retention ratio, and the scene environment feature value is multiplied by the same compression and retention ratio to obtain the weighted feature value of this channel.

[0394] When the co-discrimination gain is 0.44, divide 0.44 by 0.55 to get 0.80, and then multiply the two eigenvalues ​​of this channel by 0.80 respectively; when the co-discrimination gain is 0, the corresponding two eigenvalues ​​become 0.

[0395] Next, the second channel features are expanded in the order of the first row first and the second row last to obtain the second compressed input features containing 512 values.

[0396] The second compressed input feature is fed into the second dimensionality reduction layer. The second dimensionality reduction layer consists of 512 input nodes and 64 output nodes.

[0397] For any output node in the second dimensionality reduction layer, the 512 values ​​in the second compressed input feature are multiplied by the 512 connection weights corresponding to the output node, the products are added together, and then the bias value of the output node is added to obtain one second compressed feature value. After completing the calculation for 64 output nodes, a second compressed feature containing 64 feature values ​​is obtained.

[0398] Specifically, when there is no channel in the first channel set that satisfies a co-discrimination gain greater than 0.55, all 128 first compression feature values ​​are set to 0; when there is no channel in the second channel set that satisfies a co-discrimination gain less than or equal to 0.55, all 64 second compression feature values ​​are set to 0. Through this process, the dimensionality reduction features corresponding to different input images are guaranteed to have a fixed length.

[0399] Furthermore, batch normalization is performed on the first compression feature and the second compression feature respectively.

[0400] During batch normalization, the mean and variance of the same feature positions within the same training batch are calculated. Each feature value is subtracted from the corresponding mean, then divided by the square root of the sum of the variance and 0.000001, and then multiplied by the scaling parameter obtained during training and the offset parameter is added.

[0401] Furthermore, after batch normalization is completed, linear rectification is performed on the first and second compression features.

[0402] For eigenvalues ​​less than 0, set them to 0; for eigenvalues ​​greater than or equal to 0, keep their original values ​​unchanged.

[0403] The first compressed feature, containing 128 feature values, is arranged first, followed by the second compressed feature, containing 64 feature values. The features are then connected end-to-end according to their positions to obtain the dimensionality-reduced discriminant feature containing 192 feature values.

[0404] In addition, among the dimensionality-reduced discriminative features, the first 128 feature values ​​come from channels with a co-discrimination gain greater than 0.55, which are used to record feature responses in which both person identity information and scene environment information participate in category determination; the last 64 feature values ​​come from channels with a co-discrimination gain less than or equal to 0.55, which are used to retain feature responses that still have a classification function from a single information source.

[0405] S8 includes obtaining the dimensionality-reduced discriminant features, performing a linear transformation on the dimensionality-reduced discriminant features through a mapping matrix to obtain an initial output vector; performing nonlinear mapping and bias compensation on the initial output vector to determine the node weight vector; using a fully connected layer to perform classification mapping on the node weight vector to obtain the category probability distribution; if the maximum probability value in the category probability distribution is greater than a preset threshold, then determining the target classification result corresponding to the scene image based on the maximum probability value.

[0406] In this embodiment, the dimensionality-reduced discriminative features of the output are obtained.

[0407] Specifically, the discriminative feature is arranged in a fixed order as 192 feature values, of which the first 128 feature values ​​come from the first channel feature and the last 64 feature values ​​come from the second channel feature. The same arrangement order is used in both the training and classification phases to ensure that each feature value corresponds to the connection weight in the mapping matrix.

[0408] Furthermore, the 192 eigenvalues ​​are input into a linear mapping.

[0409] Specifically, the linear mapping layer includes 192 input nodes and 128 output nodes, and the corresponding mapping matrix contains 128 rows and 192 columns; each row of the mapping matrix corresponds to one output node, and each column corresponds to one feature position in the dimensionality-reduced discriminant features.

[0410] In this embodiment, when calculating the first output node, the first discriminant feature value is multiplied by the connection weight in the first row and first column of the mapping matrix, and the second discriminant feature value is multiplied by the connection weight in the first row and second column. This process is repeated 192 times, and the 192 products are then added together to obtain the first linear transformation value. The remaining 127 output nodes are calculated in the same way, calling the 192 connection weights in the corresponding row of the mapping matrix. After completing the calculation for all output nodes, an initial output vector consisting of 128 linear transformation values ​​is obtained.

[0411] Furthermore, the connection weights in the mapping matrix are determined during the network training phase.

[0412] Specifically, at the start of training, each connection weight is assigned an initial value according to a normal distribution with a mean of 0 and a standard deviation of 0.02. For each batch of training images input, the classification loss is calculated based on the predicted class probability and the true class label, and then the change in classification loss relative to each connection weight is calculated. This change is multiplied by the current learning rate, and the result is subtracted from the current connection weight, completing one parameter update. After training, the mapping matrix corresponding to the training epoch with the highest validation image classification accuracy is saved.

[0413] Furthermore, after the initial output vector is formed, nonlinear mapping is performed on each of the 128 linear transformation values.

[0414] Specifically, when the linear transformation value is greater than or equal to 0, the value remains unchanged; when the linear transformation value is less than 0, the value is multiplied by 0.01, and the result is used as the nonlinear mapping value of the corresponding node. This process preserves the feature response in the negative value range while limiting the magnitude influence of negative values ​​on the classification results.

[0415] In addition, each output node is set with one bias compensation value, for a total of 128 bias compensation values.

[0416] Each bias compensation value is set to 0 at the start of training and is updated synchronously according to the classification loss. The update process of the bias compensation value is the same as the update process of the connection weight, that is, the change of the classification loss relative to the corresponding bias compensation value is calculated, the change is multiplied by the current learning rate, and then the result is subtracted from the bias compensation value before the update.

[0417] In this embodiment, for each node, the nonlinear mapping value is added to the bias compensation value corresponding to that node to obtain the compensated node value.

[0418] Then, the compensated node values ​​are numerically compressed: first, the negative number of the compensated node value is taken, the natural exponent is calculated on the negative number, the resulting exponent value is added to 1, and then 1 is divided by the above addition result to obtain the node weight with a value between 0 and 1; after processing 128 nodes in sequence, a node weight vector containing 128 node weights is formed.

[0419] In this embodiment, the number of scene categories to be identified is set to 10.

[0420] Specifically, the classification fully connected layer includes 128 input nodes and 10 output nodes, with each output node corresponding to one scene category; the classification fully connected layer includes 10 sets of classification connection weights, each set containing 128 weight values, and sets one classification bias value for each category.

[0421] In determining the classification score for the first category, the 128 node weights in the node weight vector are multiplied by the weight values ​​at the same positions in the first group of classification connection weights, and then the 128 products are added together. Finally, the first classification bias value is superimposed. The classification scores for the remaining 9 categories are calculated in the same way to obtain 10 classification scores.

[0422] Furthermore, the classification connection weights and classification biases are determined using the training images.

[0423] For each training image, the category label corresponding to the real scene category is read, and the classification loss is calculated based on the 10 classification scores. The network updates the classification connection weights, classification bias values, mapping matrix and node bias compensation values ​​in reverse according to the classification loss, so that the classification score corresponding to the real scene category increases during training.

[0424] In addition, to convert the 10 category scores into category probabilities, the item with the largest value is selected from the 10 category scores. Then, the largest category score is subtracted from each category score. Natural exponentiation is performed on the 10 subtraction results, and the 10 exponentiation results are added together to obtain the total exponent value.

[0425] Then, the result of the exponent operation for each category is divided by the sum of the exponent values ​​to obtain the probability value for that category.

[0426] The maximum category score is subtracted before calculating the index value to maintain the proportional relationship between the probabilities of each category and to prevent the category score value from causing the index calculation to overflow. After the probability transformation is completed, the probabilities of all 10 categories are between 0 and 1, and the sum of the probabilities of the 10 categories is 1, thus forming the category probability distribution.

[0427] Furthermore, the 10 probability values ​​in the category probability distribution are compared, and the one with the largest value is determined as the maximum probability value. At the same time, the category number corresponding to the maximum probability value is recorded.

[0428] The maximum probability value is compared with a preset category confirmation threshold to determine whether the current scene image has a definite classification result.

[0429] Furthermore, the category confirmation threshold is determined using verification images that have not participated in network parameter updates after the neural network training is completed; 0.60, 0.65, 0.70, 0.75 and 0.80 are set as candidate thresholds in sequence, and the classification confirmation accuracy and classification confirmation ratio corresponding to each candidate threshold are calculated respectively.

[0430] Specifically, the calculation process for classification confirmation accuracy is as follows: filter the verification images whose maximum probability value is greater than the candidate threshold, count the number of images whose predicted category matches the true category, and then divide the number by the number of the selected verification images; the calculation process for classification confirmation ratio is as follows: count the number of verification images whose maximum probability value is greater than the candidate threshold, and then divide the number by the total number of verification images.

[0431] For each candidate threshold, the classification accuracy rate is multiplied by the classification accuracy rate, the product is multiplied by 2, and finally divided by the sum of the classification accuracy rate and the classification accuracy rate to obtain the evaluation value corresponding to that candidate threshold.

[0432] Among the verification results, the candidate threshold of 0.70 corresponds to the highest evaluation value, so the category confirmation threshold is set to 0.70; after network deployment, this threshold remains unchanged.

[0433] When the maximum probability value in the category probability distribution is greater than 0.70, the category corresponding to the maximum probability value is determined as the target classification result of the scene image, and the category number and its probability value are output.

[0434] When the maximum probability value is less than or equal to 0.70, no definite target classification result is generated. Instead, the scene image is marked as an image to be reviewed, and its category probability distribution is saved.

[0435] Therefore, the output of the classification result is constrained by both the category ranking result and the probability confirmation condition.

[0436] A neural network-based image classification device is also provided, including a memory and a processor; The memory is used to store programs; The processor is used to execute the program in the memory to implement the steps of the neural network-based image classification method.

[0437] In the above embodiments, the memory is used to store the image classification program, neural network parameters, and data generated during the processing.

[0438] The processor calls and executes the program to sequentially perform feature extraction, person region localization, person identity feature acquisition, scene environment feature extraction, feature fusion and classification mapping on the input scene image, and finally outputs the target classification result corresponding to the scene image.

[0439] By using the aforementioned device, the processor executes the program in the memory, enabling the separation, extraction, and collaborative fusion of person identity information and scene environment information. This reduces redundant responses and competitive interference between the two types of features, thereby improving the accuracy and stability of image classification results.

[0440] In the above embodiments, the processor can be one or more of a central processing unit, a graphics processing unit, a neural network processor, a digital signal processor, a field-programmable gate array, or an application-specific integrated circuit; the memory can be a random access memory, a read-only memory, a flash memory, a solid-state drive, or other media capable of storing programs; the processor and the memory can be integrated into the same device or can be respectively located in different devices that are interconnected.

[0441] To verify this application, experimental verification was conducted: This experiment uses ResNet-50 as the unified backbone network and implements the following reproducible comparison methods based on it: Baseline model: Standard ResNet-50 classification network, without any decoupling between people and scenes, directly performs global average pooling + fully connected classification on the entire image; Baseline + SE Attention: Add a compressed-excitation channel attention module (SE module) to the baseline model; Baseline + CBAM Attention: Add a CBAM channel-space dual attention module to the baseline model; Two-stage detection and cropping method: First, Faster R-CNN is used to detect and crop the human figure region (the bounding box is directly set to zero, without affine alignment and decay gradient masking), and then it is fed into ResNet-50 for classification. This is used to verify the difference between simply and crudely removing the human figure bounding box and the decay gradient adaptive masking removal method of this patent.

[0442] The significance of this comparison is that: SE attention and CBAM attention represent mainstream attention paradigms that do not distinguish semantic sources and only perform channel / spatial reweighting; the two-stage detection and pruning method represents a simple decoupling scheme that simply removes people without performing fine masking and preserving identity features; both can be effectively compared with the method of this patent (which simultaneously preserves identity feature discrimination information and removes noise from people in scene features).

[0443] The following two public datasets were selected to construct the experimental data: (1) MIT Indoor Scene Dataset (Indoor-67): Contains 67 indoor scene categories (classroom, restaurant, gym, office, library, etc.), with a total of 15,620 images; this dataset is widely used for indoor scene recognition benchmark tests.

[0444] To better suit the scenario of classifying complex scene images containing people, this experiment selected 24 scene categories with high human presence rates from the dataset (such as classrooms, gyms, restaurants, conference rooms, waiting rooms, and barbershops). These categories were then validated using manual annotation and a pre-trained detector. Samples with no human figures or very small human figures (less than 3% of the image area) were removed, resulting in a subset of indoor human scene images. A total of 24 categories, comprising 8192 images; The training set, validation set, and test set were divided into a 7:1.5:1.5 ratio (5734 / 1229 / 1229 images). Each image is labeled with a bounding box for the person (used as a ground truth reference for training the region proposal network branch of S2 and for evaluating the separation quality in Experiment 6).

[0445] (2) PA-100K pedestrian attribute dataset (used as a human bounding box / pose annotation auxiliary set): This dataset provides 100,000 pedestrian images and bounding box annotations in surveillance scenes, which are used to pre-train / calibrate the robustness of the human detection and affine alignment branches in S2–S3 (not directly involved in the training of the final classification task, but only used as transfer pre-training data for the region proposal network to ensure that the branch still has good human localization ability under the limited annotation of the indoor human scene subset).

[0446] Experiment 1: Comparison of Overall Classification Performance On the indoor human scene subset test set, the classification accuracy, macro-average F1 score, parameter count, and inference latency of the baseline model, baseline + SE attention, baseline + CBAM attention, two-stage detection pruning method, and the method of this application were compared. Each group was independently trained 5 times, and the results were reported as mean ± standard deviation. Paired significance test (t test) was performed on the method of this application and the best comparison method (baseline + CBAM attention).

[0447] Experimental results: Table 1. Comparison of overall classification performance of indoor character scene subset test set Significance test (comparison of the method in this application with baseline + CBAM attention, 5 independent training pairs): The difference in first-place accuracy was 7.5 percentage points, and the test probability value was less than 0.001, indicating that the difference was statistically significant.

[0448] Longitudinal comparison: From baseline model → baseline + SE attention → baseline + CBAM attention, the first hit accuracy increased by 2.7 and 1.2 percentage points respectively, indicating that simple channel / spatial attention reweighting is helpful for this task, but the improvement is limited, because they still mix the person's identity features and scene features in the same set of responses for weighting, and do not fundamentally solve the mutual interference between the two.

[0449] The two-stage detection and clipping method further improved upon the three attention methods (79.5% compared to 78.1%), indicating that physically removing the person area is indeed an effective approach, and preliminarily verifying the rationality of the design idea of ​​S5 removing the person area to obtain scene environment features; however, since this method uses rectangular hard removal and does not retain the person's identity features to participate in the final judgment, its improvement is still limited.

[0450] The method in this application further improves the accuracy by 6.1 percentage points compared to the two-stage detection and pruning method (first-hit accuracy: 85.6% compared to 79.5%), verifying the advantages of fine separation through attenuation gradient mask, global aggregation of person identity features and scene environment features, and adaptive channel compression compared to simple rectangular box culling: it avoids the loss of edge information caused by hard rectangular culling, and does not completely discard person identity information like the two-stage method, but retains it as an independent discrimination channel. Finally, it compresses redundancy through collaborative discrimination gain, so that multiple types of semantic information can be reasonably used in the same network without conflicting with each other.

[0451] In terms of cost: This patented method introduces a region proposal network and an affine alignment branch, which results in a higher number of parameters and inference time than the pure convolutional neural network method (44.3 million parameters compared to 25.6 million parameters, and 17.6 milliseconds compared to 6.1 milliseconds). However, it is still within the acceptable range for single-card real-time processing (more than 50 images can be processed per second), indicating that the accuracy gains are achieved at a reasonable computational cost and are feasible in engineering.

[0452] Experiment 2: Ablation Experiment—Module-Level Gain Decomposition Starting with the baseline model (ResNet-50), modules from each stage of the patent were progressively added, and the marginal gain of the first hit accuracy was observed to verify that each module has an independent positive contribution, thus avoiding doubts about the unclear source of the overall improvement; the configurations are defined as follows: Baseline model: No patented modules; S1 (cross-scale fusion): Only the cross-scale adaptive weighted fusion module is added to obtain an initial feature map containing spatial distribution information, while the classification head remains unchanged; Superimpose S2 and S3 (person alignment): Based on the previous step, add region proposal network localization and affine transformation cropping to obtain aligned person identity features, which are then simply concatenated with the initial feature map for classification. Overlay S4 and S5 (mask separation): Based on the previous step, add a decaying gradient mask weight map and scene environment feature removal to explicitly separate character features from scene features. Overlaying S6 and S7 (complete method): Based on the previous step, global aggregation and channel adaptive compression are added, which is the complete method of this application.

[0453] Experimental results: Table 2 Ablation Experiment: Gain of each module on classification accuracy (indoor human scene subset test set, first-position hit accuracy) S1 cross-scale fusion contributed 2.6 percentage points, indicating that cross-scale adaptive weighting based on the response differences between the candidate region of the person and the background region can indeed enable the initial feature map to better take into account both local person details and global scene semantics, providing a better feature foundation for subsequent modules.

[0454] S2 and S3 show that character alignment contributes an additional 2.6 percentage points to S1, indicating that affine transformation and cropping alignment can effectively reduce feature disturbances caused by differences in character pose, size, and shooting angle, providing a more stable character identity feature input for subsequent identity-scene separation.

[0455] S4 and S5, the single-step gain that contributes the most to mask separation (+3.7 percentage points), verify the core innovation of the patent: by adaptively determining the character region and boundary weights through the spatial decay gradient of the identity response (instead of a simple rectangle), the character identity features and scene environment features can be decoupled more precisely, reducing the classification confusion caused by semantic layer coupling. This is consistent with the conclusion in Experiment 1 that the method of this application is significantly better than the two-stage rectangle elimination method.

[0456] S6 and S7, global aggregation and channel compression contributed another 2.5 percentage points, indicating that after feature decoupling, adaptive compression of redundant channels through collaborative discriminative gain can further improve the signal-to-noise ratio of discriminative features.

[0457] Experiment 3: Training Convergence Experiment By comparing the baseline model and the method of this application under the same training configuration (learning rate, optimizer, data augmentation), the curves of the first-to-last hit accuracy of the validation set during training epochs are examined to assess convergence speed and stability.

[0458] Experimental results Table 3 Comparison of key nodes in the training convergence process (first hit accuracy on the validation set, %) like Figure 4 As shown, the method in this application consistently outperforms the baseline model throughout the entire training process, and within the first 20 training rounds, it has already achieved the accuracy level of the baseline model after convergence (66.2% compared to 74.2%). This indicates that the cross-scale adaptive weighted fusion and the decoupling mechanism between people and scenes provide the network with more discriminative feature representations, enabling the model to learn effective patterns more quickly. Because human noise in scene features is effectively suppressed, the gradient update direction is more consistent, the oscillation amplitude of the training curve is smaller than that of the baseline model, and the performance after convergence is more stable (standard deviation 0.4 compared to 0.6), thus improving classification accuracy and stability.

[0459] Experiment 4: Robustness Experiment of Person Occlusion / Posture Change Design a specific robustness experiment: Based on the indoor character scene subset test set, the character area was artificially occluded at different proportions using random rectangular block occlusion augmentation, and four occlusion intensity groups were constructed (occlusion ratio 0%–20%, 20%–40%, 40%–60%, and more than 60%, where the occlusion ratio refers to the proportion of the character's bounding box area that is occluded). The classification accuracy of the two configurations, without affine alignment and with the affine transformation alignment of this application, was compared under each occlusion group.

[0460] Experimental results: Table 4. Comparison of classification accuracy (%) under different occlusion ratios of people When the occlusion ratio is low (0%–20%), the difference between the two configurations is small (1.7 percentage points), indicating that rectangular straight cutting can basically meet the needs under light occlusion. As the occlusion ratio increases, the difference between the two curves continues to widen. When the occlusion is severe (more than 60%), the difference reaches 16.1 percentage points. This indicates that affine transformation and coordinate mapping can better preserve the relative spatial structure of the remaining visible part of the person by geometrically regularizing the bounding box of the person, and reduce the feature distortion introduced by occlusion and posture changes. This verifies the true effectiveness of the statement that the impact of the person's posture, size, shooting angle and occlusion changes on identity feature extraction is reduced. The experiment also demonstrates that the proposed method has better stability under adverse visual conditions (severe occlusion), which has engineering significance for classification tasks in real-world complex scene images (such as dense crowds and crowded indoor environments).

[0461] Experiment 5: Effectiveness Experiment of Cross-Scale Adaptive Weighted Fusion Based on the proportion of the bounding box of the person in the test set image to the total area of ​​the image, the images were divided into four scale groups (small scale: less than 5%, medium-small scale: 5%–15%, medium-large scale: 15%–30%, large scale: greater than 30%). Three feature fusion strategies were compared: single scale (using only the features of the last layer of the backbone network); fixed weight multi-scale fusion (adding features from multiple layers with equal weights, without adaptive adjustment based on response differences); and cross-scale adaptive weighted fusion as proposed in this application.

[0462] Experimental results: Table 5. Comparison of classification accuracy (%) of the three fusion strategies under different character scale proportions. All three strategies showed a common trend of "the larger the proportion of people, the more difficult the classification (the lower the accuracy)" across the four scales. This is intuitive—the larger the proportion of people, the less background information is available, and the more difficult the classification naturally becomes. Under all scale groups, the cross-scale adaptive weighted fusion proposed in this application is significantly better than single-scale feature and fixed weight fusion. The advantage is most obvious in difficult samples where people account for a large proportion (greater than 30%) (10.6 percentage points higher than single-scale and 8.1 percentage points higher than fixed weight fusion). This shows that adaptive weight allocation based on the response difference between the candidate region of people and the background region can dynamically adjust the contribution ratio of different scale features according to the specific image content. Even in difficult scenes where people dominate and background information is scarce, more effective multi-scale clues can still be extracted, reducing the information imbalance between different scale features.

[0463] Experiment 6: Character-Scene Mask Separation Quality Experiment This experiment uses manually annotated pixel-level masks of people in an indoor scene subset test set (based on bounding boxes, 500 images were sampled for fine pixel-level annotation as ground truth) as a reference to evaluate the segmentation quality of three methods for separating people regions: fixed-threshold binary mask (setting a globally fixed threshold on the response map); rectangular bounding box hard mask (i.e., the method used in the two-stage detection and cropping method); and the attenuated gradient adaptive mask (S4-S5) of this patent. Evaluation metrics include the intersection-union ratio (IoU) with the ground truth mask (the ratio of the overlapping area to the union area), the edge matching F1 score (measuring the fineness of the boundaries), and the background false positive rate (the proportion of background pixels that are falsely removed after removing people from scene environment features; the lower the better).

[0464] Experimental results: Table 6 Comparison of mask quality for three methods of character region separation (500 finely annotated samples) The intersection-over-union ratio of the rectangular bounding box hard mask is higher than that of the fixed threshold mask (the general outline of the person is enclosed), but the edge matching F1 value is significantly lower (41.2%) and the background false negative rate is the highest (27.4%). This is because the rectangular box inevitably contains a large number of non-person background pixels within the box, which are removed from the scene environment features, resulting in the loss of effective scene information. This is the fundamental reason why the two-stage detection cropping method in Experiment 1 has limited performance improvement. This patented attenuated gradient adaptive mask leads in three key metrics: an intersection-over-union ratio of 79.6%, an edge matching F1 score of 73.5% (far exceeding the 41.2% of the rectangular box), and a background false positive rate of 9.8%. This demonstrates that adaptively determining the person region and boundary weights based on the attenuated gradient of the identity response in spatial location can generate a mask that better fits the real outline of the person and has a more natural boundary gradient. While eliminating the person region, it retains the background scene information to the greatest extent. From an experimental perspective, it quantitatively verifies the effectiveness of separating the person's identity features from the scene environment features and reducing the coupling and interference between features at different semantic levels.

[0465] Experiment 7: Sensitivity Experiment to Key Hyperparameters Sensitivity analysis was performed on two key adjustable parameters in the application method: S3 confidence threshold (which determines whether to perform affine alignment and cropping on local receptive features); and S7 channel compression ratio (which determines the degree of dimensionality reduction of discriminative features). Values ​​were scanned on a subset of indoor human scene validation sets to observe changes in the first hit accuracy and determine a reasonable value range.

[0466] Experimental results: Table 7. Confidence threshold sensitivity (S3) Table 8 Channel Compression Ratio Sensitivity (S7) A confidence threshold that is too low (0.3) will force a large number of low-quality human region proposals to undergo affine alignment, introducing noise (accuracy 82.7%); a threshold that is too high (0.7) will cause a large number of real human samples with slightly low confidence to skip the alignment process, degenerating into undecoupled whole image features (accuracy drops to 82.9%). Both are worse than the intermediate value; the threshold is optimal (85.6%) when it is around 0.5, indicating that the confidence threshold design in this application has clear optimization space and a reasonable value range. The experiment supports that the default threshold setting of around 0.5 is relatively robust. The optimal compression ratio (85.6%) was achieved when the channel compression ratio was 4. If the compression ratio was too small (2, corresponding to dimension 1024), redundant information was not fully compressed. If the compression ratio was too large (16, corresponding to dimension 128), some discriminative channel information might be compressed, causing the accuracy to drop to 82.1%. This indicates that S7 requires a reasonable compression ratio for adaptive compression based on co-discriminative gain in order to achieve the expected effect of preserving discriminative contribution features and suppressing redundancy. It also shows that the compression operation is indeed adaptive and cannot achieve the same effect in any dimension. This further verifies the rationality of the co-discriminative gain compression mechanism design.

[0467] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image classification method based on neural networks, characterized in that, include: S1. Acquire scene images and perform cross-scale adaptive weighted fusion based on the response differences between candidate regions of people and background regions using a convolutional neural network to obtain an initial feature map containing spatial distribution information. S2. Based on the initial feature map containing spatial distribution information, a region proposal network is used to perform person region localization and bounding box regression to obtain local receptive features containing person bounding boxes. S3. If the confidence level of the local receptive features containing the bounding box of the person is greater than the preset confidence level threshold, then perform affine transformation and cropping operations on the local receptive features containing the bounding box of the person to obtain the aligned person identity features. S4. For the aligned identity features, the weights of the character region and boundary are adaptively determined based on the decay gradient of the identity response in the spatial location, and a mask weight map is obtained to distinguish semantic level differences. S5. Obtain the mask weight map used to distinguish semantic level differences and the initial feature map containing spatial distribution information. Remove the person region in the initial feature map containing spatial distribution information through matrix multiplication to obtain scene environment features. S6. Based on the scene environment features and the aligned character identity features, perform global aggregation processing through feature splicing operations to obtain a fused feature expression that eliminates competition interference. S7. To eliminate competition interference, the fusion feature representation is adaptively compressed based on the collaborative discriminative gain of the character identity feature and the scene environment feature in each channel to obtain the dimensionality-reduced discriminative features. S8. Based on the dimensionality-reduced discriminative features, a fully connected layer is used for classification mapping to determine the target classification result corresponding to the scene image.

2. The image classification method based on a neural network according to claim 1, characterized in that: S1 includes: The initial pixel matrix is ​​obtained by processing the scene image through a convolutional neural network, and the first response value of the candidate human region and the second response value of the background region are obtained in the initial pixel matrix. The response difference is calculated based on the first response value and the second response value. If the response difference is greater than a threshold, multiple cross-scale layers are determined. By using adaptive weights to perform cross-scale adaptive weighted fusion of multiple cross-scale layers, an initial feature map containing spatial distribution information is obtained.

3. The image classification method based on a neural network according to claim 1, characterized in that: S2 includes: Obtain an initial feature map containing spatial distribution information, generate an initial set of anchor boxes based on the initial feature map, and determine positive sample anchor boxes; Boundary box regression calculation is performed on the positive sample anchor boxes to obtain the boundary box regression offset data; By correcting the positive sample anchor boxes using bounding box regression offset data, a set of proposed boxes for the character region is obtained; Pooling is performed on the initial feature map based on the set of proposed bounding boxes for the person region to obtain local receptive features containing the bounding boxes of the person.

4. The image classification method based on a neural network according to claim 1, characterized in that: S3 includes: Obtain the local receptive features containing the bounding box of the person and the confidence level of the local receptive features; If the confidence level is greater than the preset confidence level threshold, the affine transformation matrix is ​​determined based on the coordinates of the character's bounding box. The local receptive features are mapped to spatial coordinates using an affine transformation matrix to obtain the mapped feature pixel distribution. The cropping operation region is determined in the mapped feature pixel distribution and the cropping operation is performed to obtain the aligned human identity features.

5. The image classification method based on a neural network according to claim 1, characterized in that: S4 includes: Obtain the aligned identity features of the individuals, extract the identity responses of the identity features, and obtain the response distribution matrix; The numerical difference is calculated based on the response distribution matrix to obtain the set of decay gradients of the identity response; The region segmentation result is determined based on the set of decay gradients. The region segmentation result is used to determine the character region and assign boundary weights to obtain semantic hierarchical features with boundary weights. Based on semantic hierarchical features, hierarchical differences are distinguished, and a mask weight map is obtained to distinguish semantic hierarchical differences.

6. The image classification method based on a neural network according to claim 1, characterized in that: S5 includes: Obtain the mask weight map and the initial feature map containing spatial distribution information; The mask weight map is processed by feature dimension transformation to obtain the alignment weight matrix; Extract pixel values ​​of the character region based on the alignment weight matrix, and generate an inverse mask matrix; If the element value of the inverse mask matrix is ​​greater than the preset threshold, the region corresponding to the element value is determined to be the background pixel retention region, and the background mask image is obtained. The background mask image is multiplied by the initial feature image to remove the character area from the initial feature image, thus obtaining the scene environment features.

7. The image classification method based on a neural network according to claim 1, characterized in that: S6 includes: Obtain scene environment features and character identity features, and perform dimensional alignment processing on scene environment features and character identity features to obtain aligned features; Semantic mapping is performed based on alignment features to obtain the association feature matrix; The correlation feature matrix is ​​subjected to interference filtering to determine the pure feature vector, which is generated by adaptive pooling operation; The pure feature vectors are subjected to feature concatenation and global aggregation to obtain a fused feature representation that eliminates competitive interference.

8. The image classification method based on a neural network according to claim 1, characterized in that: S7 includes: Obtain the fusion feature representation after eliminating competitive interference, extract the character identity features and scene environment features from the fusion feature representation, and calculate the initial activation intensity; Calculate the collaborative discrimination gain based on the initial activation strength; If the collaborative discrimination gain is greater than a preset threshold, the first channel feature is obtained; if the collaborative discrimination gain is less than or equal to the preset threshold, the second channel feature is obtained. Adaptive compression is performed on the first channel features and the second channel features based on the co-discriminative gain to obtain the dimensionality-reduced discriminative features.

9. The image classification method based on a neural network according to claim 1, characterized in that: S8 includes: Obtain the dimensionality-reduced discriminant features, and perform a linear transformation on the dimensionality-reduced discriminant features through a mapping matrix to obtain the initial output vector; Nonlinear mapping and bias compensation are performed on the initial output vector to determine the node weight vector; A fully connected layer is used to classify and map the node weight vectors to obtain the category probability distribution; If the maximum probability value in the category probability distribution is greater than the preset threshold, the target classification result corresponding to the scene image is determined based on the maximum probability value.

10. An image classification device based on a neural network, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program in the memory to implement the steps of the neural network-based image classification method as described in any one of claims 1-9.

Images