Model training method and device and target equipment identification method

Abstract

The invention discloses a model training method and device and a target equipment identification method. The method comprises the following steps: acquiring a plurality of image frames in a video stream; performing convolution operation on the image frame, and performing batch normalization processing on features obtained by the convolution operation to obtain a first target feature map; performing convolution operation on the first target feature map by using convolution kernels of different scales, and fusing output feature maps of the convolution kernels of different scales to obtain n layers of second target feature maps output by the convolution layers; performing multi-scale feature fusion on the second target feature map of each layer in the n layers of second target feature maps to obtain a first feature corresponding to the second target feature map of each layer, and performing weighted fusion on the first features to obtain a third target feature map; and training the deep learning model by using the third target feature map to obtain a target equipment identification model. The technical problem that the recognition precision of a related target equipment recognition model is low is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision application technology, and more specifically, to a model training method, device, and target equipment recognition method. Background Art

[0002] Most existing recognition systems for target equipment such as helmets are based on traditional computer vision technologies, which usually rely on hand-crafted feature extraction methods and static target detection models. Although these methods perform well in some scenarios, their performance is often limited when faced with diverse practical application environments. Specifically, false detection or missed detection may occur in situations with diverse human postures and a large number of target equipment types.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a model training method, an apparatus, and a target equipment recognition method to at least solve the technical problem of low recognition accuracy of the relevant target equipment recognition model.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, including: acquiring multiple image frames in a video stream, wherein each image frame includes: a target object wearing a target device; performing a convolution operation on the image frames, and performing batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map; performing a convolution operation on the first target feature map using convolution kernels of different scales, and fusing the output feature maps of the convolution kernels of different scales to obtain a second target feature map of n layers output by the convolution layer, wherein n is a positive integer greater than 1; performing multi-scale feature fusion on the second target feature map of each layer in the second target feature map of the n layers to obtain a first feature corresponding to the second target feature map of each layer, and performing weighted fusion on the first features to obtain a third target feature map; and training a deep learning model using the third target feature map to obtain a target device recognition model, wherein the target device recognition model is used to recognize whether the target object is wearing the target device.

[0006] Optionally, a convolution operation is performed on the image frame, and features obtained by the convolution operation are batch normalized to obtain a first target feature map, including: in each residual block of the backbone network, a convolution operation is performed on the image frame, and features obtained by the convolution operation are batch normalized to obtain a first feature map; in each residual block of the backbone network, a 1×1 convolution kernel is used to convolve the first feature map, an n×n convolution kernel is used to extract features after the convolution operation, and a 1×1 convolution kernel is used to convolve the extracted features to obtain a second feature map, wherein n is a positive integer greater than 1; a preset activation function is used to activate the second feature map to obtain a third feature map; and the third feature map is added to the image frame to obtain the first target feature map.

[0007] Optionally, multi-scale feature fusion is performed on the second target feature map of each layer in the n layers to obtain the first feature corresponding to the second target feature map of each layer, including: for the second target feature map of the i-th layer, upsampling the second target feature map of the i+1-th layer, and fusing the upsampling result with the i-th second target feature map to obtain a first fusion result, where i is a positive integer not greater than n; for the second target feature map of the i-th layer, downsampling the second target feature map of the i-1-th layer, and fusing the downsampling result with the second target feature map of the i-th layer to obtain a second fusion result; for the second target feature map of the i-th layer, fusing the first fusion result and the second fusion result to obtain the first feature corresponding to the second target feature map of each layer.

[0008] Optionally, weighted fusion is performed on the first features to obtain a third target feature map, including: determining the weight of each first feature according to a first parameter and a second parameter corresponding to each first feature, wherein the first parameter is used to characterize the expressive power of the first feature, and the second parameter is used to characterize the contribution of the first feature to the recognition task; and weighted fusion is performed on the first features according to the weight of each first feature to obtain the third target feature map.

[0009] Optionally, the first parameter includes at least information entropy; the second parameter includes at least mutual information, wherein the mutual information is used to characterize the correlation between the feature map output by the deep learning model and the preset task label.

[0010] Optionally, the third target feature map is used to train the deep learning model to obtain a target equipment recognition model, including: using the prediction function of the center point of the regression network to predict the coordinates of the center point of the target equipment in the third target feature map; using the prediction function of the size of the regression network to predict the size of the target equipment in the third target feature map; calculating the position and bounding box of the target equipment based on the predicted coordinates of the center point of the target equipment and the size of the target equipment to obtain a model output result; calculating a first error between the predicted coordinates of the center point of the target equipment and the coordinates of the actual center point, and calculating a second error between the predicted size of the target equipment and the actual size; performing weighted summation of the first error and the second error to obtain a loss function; using the third target feature map to train the deep learning model, and when the loss function meets preset requirements, obtaining the target equipment recognition model.

[0011] Optionally, before performing a convolution operation and batch normalization on the image frame to obtain the first target feature map, the method also includes: using a median filter or a Gaussian filter to remove noise in the image frame; normalizing the pixel values ​​in the image frame to the range of [0, 1], and adjusting the size of the image frame to the target size required by the model input; randomly cropping the image frame, and rotating the image frame at a random angle.

[0012] According to another aspect of an embodiment of the present application, a method for identifying a target equipment is also provided, including: obtaining an image to be identified, wherein the image to be identified includes: a target object; using a target equipment identification model to identify whether the target object in the image to be identified is wearing the target equipment, wherein the target equipment identification model is trained by the above-mentioned model training method.

[0013] According to another aspect of the embodiment of the present application, a model training device is also provided, including: an acquisition module, used to acquire multiple image frames in a video stream, wherein each image frame includes: a target object wearing a target equipment; a first processing module, used to perform a convolution operation on the image frame, and perform batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map; a second processing module, used to perform a convolution operation on the first target feature map using convolution kernels of different scales, and fuse the output feature maps of the convolution kernels of different scales to obtain an n-layer second target feature map output by the convolution layer, wherein n is a positive integer greater than 1; a fusion module, used to perform multi-scale feature fusion on the second target feature map of each layer in the n-layer second target feature map, obtain a first feature corresponding to the second target feature map of each layer, and perform weighted fusion on the first feature to obtain a third target feature map; a training module, used to train a deep learning model using the third target feature map to obtain a target equipment recognition model, wherein the target equipment recognition model is used to identify whether the target object is wearing the target equipment.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, and the storage medium includes a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the above model training method.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above model training method is executed when the program is running.

[0016] According to another aspect of the embodiments of the present application, a computer program is also provided, wherein the above model training method is implemented when the computer program is executed by a processor.

[0017] According to another aspect of the embodiments of the present application, a computer program product is also provided, which includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above model training method is implemented.

[0018] In an embodiment of the present application, a plurality of image frames in a video stream are obtained, wherein each image frame includes: a target object wearing a target equipment; a convolution operation is performed on the image frame, and features obtained by the convolution operation are batch normalized to obtain a first target feature map; a convolution operation is performed on the first target feature map using convolution kernels of different scales, and output feature maps of the convolution kernels of different scales are fused to obtain a second target feature map of n layers output by the convolution layer, wherein n is a positive integer greater than 1; multi-scale feature fusion is performed on the second target feature map of each layer in the second target feature map of the n layers to obtain a first feature corresponding to the second target feature map of each layer, and weighted fusion is performed on the first feature to obtain a third target feature map; a deep learning model is trained using the third target feature map to obtain a target equipment recognition model, wherein the target equipment recognition model is used to recognize whether the target object is wearing the target equipment, thereby achieving the purpose of improving the efficiency of feature extraction and the convergence speed of the model, thereby achieving the technical effect of improving the recognition accuracy of the target equipment recognition model, and further solving the technical problem of low recognition accuracy of the relevant target equipment recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flow chart of a model training method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of another model training method according to an embodiment of the present application;

[0022] Figure 3 is a structural diagram of a model training device according to an embodiment of the present application;

[0023] Figure 4 It is a hardware structure block diagram of a computer terminal according to a model training method of an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to an embodiment of the present application, a method embodiment of a model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] Figure 1 is a flow chart of a model training method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0028] Step S101: acquiring a plurality of image frames in a video stream, wherein each image frame includes: a target object wearing a target device.

[0029] Among them, target equipment includes: helmets, safety belts, reflective clothing / vests, goggles, etc.

[0030] Image frames are continuously extracted from the video stream, each of which may contain a target object, such as an electric bike rider. These image frames will be the basis for subsequent feature extraction and model training. The image frames are usually obtained by real-time capture using a smart camera or extracted from a pre-recorded video.

[0031] Step S102, performing a convolution operation on the image frame, and performing batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map.

[0032] First, a convolution operation is applied to each image frame. The convolution operation can capture local features in the image, such as edges, textures, etc. Next, the feature map obtained by the convolution operation is batch normalized to improve the stability of training and accelerate convergence. Batch normalization standardizes the values ​​in the feature map and reduces internal covariate shift, thereby optimizing the model training process.

[0033] Step S103, using convolution kernels of different scales to perform a convolution operation on the first target feature map, and fusing the output feature maps of the convolution kernels of different scales to obtain a second target feature map of the n-layer output of the convolution layer, where n is a positive integer greater than 1.

[0034] Convolution operations are performed on the first target feature map after batch normalization using convolution kernels of various sizes to capture multi-scale feature information. Convolution kernels of different sizes can capture different details of the target. For example, a 3x3 convolution kernel may focus on capturing detail features, while a 7x7 convolution kernel is more suitable for extracting overall shape features. The output feature maps of these convolution kernels of different sizes are fused to build a more comprehensive and richer feature representation, which helps the model achieve better performance in the subsequent recognition process.

[0035] Specifically, you can choose a variety of convolution kernels of different sizes, such as 3x3, 5x5, and 7x7. These convolution kernels are used to extract features of different scales from the input image. Apply convolution kernels of each size to the input feature map to obtain multiple feature maps of different sizes. The output feature maps of convolution kernels of different sizes are fused. The fusion method can be concatenation or weighted summation of feature maps. Finally, the fused feature map is input to the subsequent network layer for further processing and analysis. This step helps the network to utilize the multi-scale information extracted by different convolution kernels and improve the detection ability of objects of different scales.

[0036] Step S104, performing multi-scale feature fusion on the second target feature map of each layer in the n layers of second target feature maps to obtain the first features corresponding to the second target feature map of each layer, and performing weighted fusion on the first features to obtain a third target feature map.

[0037] The obtained n-layer second target feature map is subjected to multi-scale feature fusion. The purpose of this step is to further enhance the model's ability to detect targets of different scales. Then, the fused features are weighted fused, and the weights of feature maps at different levels are adaptively adjusted so that important features can be more fully expressed in the final fused third target feature map, thereby improving the detection accuracy and robustness of the model.

[0038] Step S105: Use the third target feature map to train the deep learning model to obtain a target equipment recognition model, wherein the target equipment recognition model is used to identify whether the target object is wearing the target equipment.

[0039] By using the third target feature map, which has been processed by multi-scale feature fusion and weighted fusion, a deep learning model, namely the target equipment recognition model, is trained. By learning the association between these feature maps and target equipment (such as a helmet), the model can determine whether the target object is wearing the target equipment. Through a large amount of training data and iterative optimization, the model will continuously adjust its parameters to improve the accuracy and real-time performance of recognition.

[0040] According to the above steps, a plurality of image frames in a video stream are obtained, wherein each image frame includes: a target object wearing a target device; a convolution operation is performed on the image frame, and features obtained by the convolution operation are batch normalized to obtain a first target feature map; a convolution operation is performed on the first target feature map using convolution kernels of different scales, and output feature maps of the convolution kernels of different scales are fused to obtain a second target feature map of n layers output by the convolution layer, wherein n is a positive integer greater than 1; multi-scale feature fusion is performed on the second target feature map of each layer in the second target feature map of the n layers to obtain a first feature corresponding to the second target feature map of each layer, and weighted fusion is performed on the first features to obtain a third target feature map; the third target feature map is used to train a deep learning model to obtain a target device recognition model, wherein the target device recognition model is used to recognize whether the target object is wearing the target device, thereby achieving the purpose of improving the efficiency of feature extraction and the convergence speed of the model, thereby realizing the technical effect of improving the recognition accuracy of the target equipment recognition model.

[0041] The following Figure 1 The steps shown are exemplary and explanatory.

[0042] According to some optional embodiments of the present application, a convolution operation is performed on an image frame, and features obtained by the convolution operation are batch normalized to obtain a first target feature map. This can be achieved by the following method: in each residual block of the backbone network, a convolution operation is performed on the image frame, and features obtained by the convolution operation are batch normalized to obtain a first feature map; in each residual block of the backbone network, a 1×1 convolution kernel is used to convolve the first feature map, an n×n convolution kernel is used to extract features after the convolution operation, and a 1×1 convolution kernel is used to convolve the extracted features to obtain a second feature map, where n is a positive integer greater than 1; a preset activation function is used to activate the second feature map to obtain a third feature map; and the third feature map is added to the image frame to obtain the first target feature map.

[0043] At the beginning of each residual block, a convolution operation is performed on the input image frame or the feature map of the previous stage. The convolution operation can capture local features in the image, such as edges, textures, etc., which is the basis for deep learning models to recognize objects. After the convolution operation, batch normalization is used to standardize the output of the convolution layer, reduce internal covariate shift, and improve the training stability and speed of the model. The specific formula for batch normalization is as follows:

[0044]

[0045] Where F is the convolution output, μ and σ are the mean and variance in the batch, γ and β are learnable parameters, and ∈ is a tiny constant to prevent division by zero.

[0046] In the residual block, a bottleneck structure is used for feature extraction. The bottleneck structure first uses a 1×1 convolution kernel to reduce the dimension of the input feature, then uses an n×n convolution kernel (n>1, such as 3×3) for feature extraction, and finally restores the feature size through a 1×1 convolution kernel. The purpose of this process is to enhance the feature expression capability of the convolution layer without significantly increasing the amount of calculation. The specific formula is as follows:

[0047] Bottleneck(I)=Conv(ReLU(BN(Conv1r1(I))),3r3)+Conv1

[0048] Among them, Conv1 and Conv3x represent 1x1 and 3x3 convolution operations respectively.

[0049] After feature extraction, the feature map is activated again using a preset activation function (such as ReLU) to enhance the nonlinear expression capability. Subsequently, the activated feature map is added to the input image frame or the feature map of the previous stage. This is the core idea of ​​the residual connection. The residual connection makes it easier for the model to learn residual information, thereby accelerating the training process and improving the final detection accuracy. The output of the residual connection and activation is the first target feature map, and the formula is as follows:

[0050] O=ReLU(BN(Conv1z1(ReLU(BN(ConV33(I)))))+Conv1c1(I)

[0051] Among them, (O) is the first target feature map of the output, and (I) is the input feature map.

[0052] Through this series of processing, each residual block can efficiently extract and fuse the feature information in the image, and the final first target feature map will be used as the input of the subsequent module for more complex target detection tasks. This design method not only improves the model's ability to capture image features, but also optimizes the model's training process through batch normalization, bottleneck structure, and residual connection technologies, reduces the gradient vanishing problem, and enables the model to be built more deeply, thereby achieving better results in detecting whether electric bike riders are wearing helmets.

[0053] According to some other optional embodiments of the present application, multi-scale feature fusion is performed on the second target feature map of each layer in the second target feature map of each layer to obtain the first feature corresponding to the second target feature map of each layer, which can be achieved by the following method: for the second target feature map of the i-th layer, the second target feature map of the i+1-th layer is upsampled, and the up-sampling result is fused with the i-th second target feature map to obtain a first fusion result, wherein i is a positive integer not greater than n; for the second target feature map of the i-th layer, the second target feature map of the i-1-th layer is downsampled, and the down-sampling result is fused with the second target feature map of the i-th layer to obtain a second fusion result; for the second target feature map of the i-th layer, the first fusion result and the second fusion result are fused to obtain the first feature corresponding to the second target feature map of each layer.

[0054] For the second target feature map of the i-th layer (referred to as Fi), the upsampling operation starts from the second target feature map (Fi+1) of the i+1th layer, and its goal is to restore the feature map of the higher layer to the resolution of the lower layer so as to be fused with Fi. Upsampling can be achieved by bilinear interpolation, nearest neighbor interpolation, or other upsampling techniques. Once Fi+1 is upsampled to the same resolution as Fi, the two feature maps can be fused by concatenation or weighted summation to obtain a first fusion result. Similarly, for the second target feature map Fi of the i-th layer, a downsampling operation can also be performed from the second target feature map (Fi-1) of the i-1th layer. The goal of downsampling is to reduce the resolution of the feature map and fuse it with Fi to enhance the detection of smaller-scale objects. Downsampling can be achieved by pooling operations or convolutions with larger strides. After downsampling Fi-1 to the same resolution as Fi, the two feature maps are fused to obtain a second fusion result. Finally, the first fusion result and the second fusion result are fused to obtain the first feature corresponding to the second target feature map of each layer.

[0055] In some optional embodiments of the present application, weighted fusion of the first features is performed to obtain a third target feature map, which can be achieved by the following method: determining the weight of each first feature based on the first parameter and the second parameter corresponding to each first feature, wherein the first parameter is used to characterize the expressive power of the first feature, and the second parameter is used to characterize the contribution of the first feature to the recognition task; performing weighted fusion of the first features based on the weight of each first feature to obtain the third target feature map.

[0056] The first parameter is used to evaluate the expressiveness of the first feature. A feature map with strong expressiveness can capture the features of the target object more clearly and completely. In deep learning, expressiveness can be measured by the distribution of activation values ​​of the feature map, for example, the proportion of non-zero elements after the ReLU activation function on the feature map, or the entropy value of the feature map. The higher the entropy value, the greater the uncertainty of the information in the feature map, that is, the stronger the expressiveness. The second parameter is used to evaluate the contribution of the first feature to the recognition task. Features in the feature map that are highly relevant to the recognition task will contribute more to the final detection effect. During the model training process, the second parameter can be obtained by calculating the correlation or weight between the feature map and the recognition result. For example, the correlation calculation between the feature map and the loss function specific to the recognition task. For example, the contribution of the feature map to the loss function or the gradient size can be used as a reference for the second parameter. Combining the first parameter and the second parameter, the weight of each first feature can be determined. The weight not only reflects the expressiveness of the feature map, but also takes into account its actual contribution to the recognition task. Using the weight of each first feature, weighted fusion is performed to obtain the third target feature map. Weighted fusion takes into account the actual contribution of features at different levels, so that the fused feature map can more effectively capture the key information of the target object. Through the above steps, the model can dynamically adjust the weights in the feature fusion process according to the expressive power of multi-scale features and their actual contribution to the recognition task, thereby constructing a more efficient and accurate feature representation, which is suitable for recognition tasks such as whether electric bike riders are wearing helmets, and improves detection accuracy and robustness.

[0057] In other words, each layer of feature maps is assigned a weight. These weights are learned to adaptively adjust the contribution of feature maps at different levels. Each layer of feature maps is adaptively weighted so that important feature maps have a larger weight in the final fusion. The adaptive mechanism adjusts the weights based on the expressiveness of the feature maps and their contribution to the final task. The weighted feature maps are summed to obtain the fused feature map. In this way, the feature information of low and high layers can be integrated to improve the model's perception of objects of different scales.

[0058] Preferably, the first parameter includes at least: information entropy; the second parameter includes at least: mutual information, wherein the mutual information is used to characterize the correlation between the feature map output by the deep learning model and the preset task label. Information entropy is an indicator to measure the uncertainty or amount of information. In deep learning, it can be used to evaluate the expressiveness of the feature map. A higher information entropy means that the information contained in the feature map is more uncertain or complex, which usually means that the feature map can better capture the diversity and details in the image. Mutual information is an indicator to measure the correlation between two random variables, and is used to evaluate the dependency between the feature map and the task label. In deep learning, mutual information can help the model understand the actual contribution of the feature map to the recognition task.

[0059] As some optional embodiments of the present application, the deep learning model is trained using the third target feature map to obtain a target equipment recognition model, which can be achieved by the following method: using the prediction function of the center point of the regression network to predict the coordinates of the center point of the target equipment in the third target feature map; using the prediction function of the size of the regression network to predict the size of the target equipment in the third target feature map; according to the predicted coordinates of the center point of the target equipment and the size of the target equipment, the position and bounding box of the target equipment are calculated to obtain the model output result; calculating the first error between the predicted coordinates of the center point of the target equipment and the coordinates of the actual center point, and calculating the second error between the predicted size of the target equipment and the actual size; performing weighted summation of the first error and the second error to obtain a loss function; using the third target feature map to train the deep learning model, and when the loss function meets the preset requirements, the target equipment recognition model is obtained.

[0060] In order to improve the accuracy of the electric vehicle without helmet detection system, the loss function needs to be optimized to better meet the needs of the target detection task. The design of the loss function takes into account the position and size of the target to comprehensively evaluate the quality of the model prediction. The position loss is used to measure the error between the predicted target center point and the true target center point. The prediction of the target center point is the key to the Anchor-Free detection method. The position loss formula is as follows:

[0061]

[0062] Where: (x c,i ,y c,i ) is the true center point coordinate of the i-th target, (x^ c,i ,y^ c,i ) is the predicted center point coordinate of the ith target, and N is the total number of targets.

[0063] The position loss is calculated as the sum of the squares of the Euclidean distances between the predicted center point and the true center point, reflecting the accuracy of the predicted position.

[0064] The size loss is used to measure the difference between the predicted target size (width and height) and the actual target size. The size loss formula is as follows:

[0065]

[0066] Among them, (w i ,h i ) is the true size (width and height) of the i-th object, (w^ i ,h^ i ) is the predicted size (width and height) of the i-th object and N is the total number of objects.

[0067] The size loss calculates the square difference between the predicted size and the true size, which measures the accuracy of the target size prediction. Finally, the position loss and size loss are combined and weighted using a weight coefficient to achieve a balance between different tasks. By introducing an improved loss function, the system can better balance the accuracy of position and size prediction, thereby improving the overall effect of electric vehicle helmet wearing detection. Comprehensive consideration of the center point and size of the target ensures the accuracy and reliability of the detection results, thereby enhancing the real-time monitoring capability of the system.

[0068] In some optional embodiments of the present application, before performing a convolution operation and a batch normalization process on the image frame to obtain the first target feature map, the following steps may also be performed:

[0069] Use median filtering or Gaussian filtering to remove noise from the image frame; normalize the pixel values ​​in the image frame to the range of [0, 1], and adjust the size of the image frame to the target size required by the model input; randomly crop the image frame, and rotate the image frame at random angles.

[0070] Use median filtering or Gaussian filtering to remove noise from image frames to reduce the impact of image noise on subsequent feature extraction and detection. Median filtering is a common nonlinear filtering method that removes salt and pepper noise from images by sorting pixel values ​​in a local area of ​​the image and taking the median. Median filtering is not as effective as Gaussian filtering in removing Gaussian noise, but it is effective in dealing with salt and pepper noise or other isolated point noise. Gaussian filtering uses a Gaussian kernel to perform convolution on an image to smooth the image and reduce the impact of random noise. Gaussian filtering is effective for Gaussian noise, and can also better maintain the edges and details of the image.

[0071] Normalize the pixel values ​​in the image frame to the range of [0, 1] to ensure that the input data of the model is on the same scale and avoid numerical instability caused by too large or too small input data range. Adjust the image frame to the target size required by the model input, such as 640×640 pixels. Resizing ensures that the model can process the input data efficiently, unify the data input standard, and avoid the additional computational burden caused by size mismatch. Resizing can be achieved by the following steps: determine the target size; scale the image using bilinear interpolation, nearest neighbor interpolation, or other interpolation methods. To improve the generalization ability of the model, the image frame can be randomly cropped and rotated. Randomly select a region from the image frame and keep or discard this region to generate a new image sample. Random cropping helps the model learn the representation of the target at different positions and scales, and enhances the robustness of the model to position changes. Rotate the image frame by random angles to generate new image samples. The rotation angle can be randomly selected to ensure that the model can handle changes in the target in different directions and improve the direction invariance of the model.

[0072] The embodiment of the present application also provides a method for identifying a target device, comprising: obtaining an image to be identified, wherein the image to be identified includes: a target object; using a target device identification model to identify whether the target object in the image to be identified is wearing the target device, wherein the target device identification model is Figure 1 The model is trained using the model training method.

[0073] It should be noted that the preferred implementation of the above embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0074] Figure 2 is a flowchart of another model training method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0075] Step S201, the smart camera acquires the image to be identified. A high-resolution smart camera is used to capture the real-time video stream of the electric vehicle. The video stream transmitted by the camera is sent to the image processing module, and the image data is processed in real time by the OpenCV library to extract continuous image frames as detection input.

[0076] Step S202: image preprocessing and data enhancement.

[0077] Step S2021, image preprocessing: denoising, normalizing and resizing the acquired image. Specifically, it includes: using median filtering or Gaussian filtering to remove noise in the image. Normalizing the pixel values ​​to the range of [0, 1]. Adjusting the image size to the size required by the model input (such as 640×640).

[0078] Step S2022: To improve the generalization ability of the model, data enhancement is performed on the image, including: randomly cropping areas in the image, rotating the image at random angles, and adjusting the brightness and contrast of the image.

[0079] Step S203, using the improved ResNet structure as the backbone network.

[0080] In the electric vehicle helmet-less detection system, an improved version of the ResNet structure (ResNetV2) is used as the backbone network of the model. The main improvement of ResNetV2 is the residual block.

[0081] Step S2031, in each residual block, ResNetV2 introduces batch normalization after the convolution operation, which helps to standardize the features so that their mean is 0 and their variance is 1. The formula is as follows:

[0082]

[0083] Where F is the convolution output, μ and σ are the mean and variance in the batch, γ and β are learnable parameters, and ∈ is a tiny constant to prevent division by zero.

[0084] Step S2032 introduces a "bottleneck" structure, which first performs a 1x1 convolution to reduce the number of channels, then performs a 3x3 convolution to extract features, and finally restores the number of channels through a 1x1 convolution. This structure helps improve the efficiency of the model by reducing the amount of calculation without losing information. The calculation process is as follows:

[0085] Bottleneck(I)=Conv(ReLU(BN(Conv1c1(I))),3c3)+Conv1

[0086] Among them, Conv1 and Conv3x represent 1x1 and 3x3 convolution operations respectively.

[0087] Step S2033, in each residual block, the convolution output after batch normalization and ReLU activation is added to the input I, the formula is:

[0088] O=ReLU(BN(Conv1c1(ReLU(BN(ConV33(I)))))+Conv1c1(I)

[0089] The traditional ResNet mainly uses the basic residual block structure and ReLU activation function, while the improved network ResNetV2 improves the model's expressiveness and computational efficiency by introducing batch normalization, bottleneck structure, and improved residual connection methods. ResNetV2 improves the accuracy of feature extraction and the convergence speed of the model, making the model perform better in real-time detection of whether an electric bike is wearing a helmet.

[0090] Step S204, introduce mixed convolution for feature extraction. In the electric vehicle without helmet detection system, the mixed convolution (MixConv) method is used to improve feature extraction. This method enhances the feature extraction capability by fusing convolution kernels of different sizes, so that the model can capture multi-scale feature information. The following is a detailed description of the process:

[0091] Step S2041, convolution operation.

[0092] Define convolution kernels of different sizes: Select convolution kernels of different sizes, such as 3x3, 5x5, and 7x7. These convolution kernels are used to extract features of different scales from the input image. Apply convolution kernels: Apply convolution kernels of each size to the input feature map I to obtain multiple feature maps of different sizes. For a specific size of convolution kernel k (for example, k = 3 corresponds to a 3x3 convolution kernel), the convolution operation can be expressed as:

[0093] I k =I*k

[0094] Among them, * represents the convolution operation, I k It is the feature map obtained after convolution.

[0095] Step S2042, feature fusion.

[0096] The output feature maps of convolution kernels of different sizes are fused. The fusion method can be concatenation or weighted summation of feature maps. Assume that there are n types of convolution kernels and the output feature map is I k1 ,I k2 ,…,I kn , then the fused feature map I mix It can be expressed as:

[0097]

[0098] Among them, α i is the fusion weight, which reflects the importance of each convolution kernel feature map in the final feature map.

[0099] The fused feature map is input into the subsequent network layer for further processing and analysis. This step helps the network to utilize the multi-scale information extracted by different convolution kernels and improve the detection ability of objects of different scales.

[0100] Traditional convolutional neural networks usually use fixed-size convolution kernels for feature extraction, which may be difficult to process multi-scale feature information. By introducing mixed convolution (MixConv), the model can use convolution kernels of different sizes to extract multi-scale features and fuse these feature maps to enhance the detection ability of targets of various scales. The mixed convolution method improves the richness and accuracy of feature extraction, allowing the model to better identify and distinguish targets of different scales when detecting whether an electric vehicle is wearing a helmet in real time.

[0101] Step S205, using an improved feature pyramid network. In the electric vehicle without helmet detection system, the ability of the feature pyramid network is improved by using an improved FPN (such as PANet) to enhance the model's detection accuracy for targets of different scales. The specific improvement process is as follows:

[0102] Step S2051, feature fusion mechanism. By introducing bottom-up and top-down feature fusion mechanisms, the fusion process of multi-scale features is optimized. PANet improves the simple feature fusion method in traditional FPN and adds a bottom-up feature fusion step to enhance the information transfer and fusion capabilities between feature maps.

[0103] Top-down feature fusion: transfer information from high-level feature maps to low-level feature maps. For the feature map Li of the i-th layer, the top-down fusion process includes upsampling the feature map of the i+1th layer and fusing it with the feature map of the current layer Li. The fused feature map is then subjected to a convolution operation. The formula is as follows:

[0104]

[0105] Among them, Conv is the convolution operation, ReLU is the activation function, and Upsample(Li+1) is the operation of upsampling the feature map of the i+1th layer.

[0106] Bottom-up feature fusion: transfer information from low-level feature maps to high-level feature maps. For the feature map Li of the i-th layer, the bottom-up fusion process includes downsampling the feature map of the i-1th layer and fusing it with the feature map of the current layer Li. The fused feature map is then subjected to a convolution operation. The formula is as follows:

[0107]

[0108] Among them, Downsample(Li-1) is the operation of downsampling the feature map of the i-1th layer.

[0109] Step S2052, multi-scale feature representation. Through the improved feature fusion mechanism, feature representations of multiple scales can be effectively generated. These multi-scale features enhance the detection capability of targets of different scales by integrating low-level and high-level information. For example, for the detection of electric vehicles without helmets, PANet can better process the features of helmets of different sizes and the electric vehicle body, thereby improving the detection accuracy.

[0110] Traditional FPN methods usually use top-down feature fusion, but may have limited effect when processing multi-scale features. PANet, by introducing a bottom-up feature fusion mechanism, enhances the fusion capability between feature maps at different levels and improves the detection accuracy of targets of various scales. The improvement of PANet improves the model's ability to process multi-scale features, making the real-time detection of electric vehicles without helmets more accurate.

[0111] Step S206, improve cross-layer feature fusion. In the electric vehicle without helmet detection system, in order to further improve the detection capability of multi-scale targets, a cross-layer feature fusion module (ASFF, Adaptive Spatial Feature Fusion) is introduced. The specific process is as follows:

[0112] Step S2061, based on the traditional FPN, an adaptive weight mechanism is introduced to optimize the contribution of feature maps of different scales. The weight w of each layer of feature map i It is calculated adaptively and aims to improve the effectiveness of each layer of features in the fusion process.

[0113] The calculation formula of feature weighting: feature map L of each layer i will be assigned a weight w i These weights are obtained through learning to adaptively adjust the contribution of feature maps at different levels. The fused feature map F fuse It can be expressed as the following weighted sum:

[0114]

[0115] Among them, F fuse is the fused feature map, w i is the weight of the feature map of layer iii, indicating the importance of the feature map of this layer in the fusion. i is the feature map of the i-th layer.

[0116] Step S2062, feature fusion mechanism. Adaptive adjustment of feature maps: adaptively weight the feature maps of each layer so that important feature maps occupy a larger weight in the final fusion. The adaptive mechanism adjusts the weights according to the expressive power of the feature maps and their contribution to the final task. Weighted fusion: sum the weighted feature maps to obtain a fused feature map. In this way, the feature information of low and high layers can be integrated to improve the model's perception of targets of different scales. In traditional FPN, feature fusion is usually achieved by simple summation of feature maps, without considering the adaptive weights of feature maps at different levels. ASFF optimizes the contribution of feature maps by introducing an adaptive weight mechanism, making cross-layer feature fusion more accurate and able to better handle targets of various scales. The model's ability to handle multi-scale features has been improved, making the detection of electric vehicles without helmets more accurate and robust.

[0117] Step S207, improve the Anchor-Free detection. In the electric vehicle without helmet detection system, an improved Anchor-Free detection method (CenterNet) is introduced to improve the detection accuracy and simplify the detection process. The specific improvement process is as follows:

[0118] Step S2071: Anchor-Free detection method directly predicts the center point position of the target through a regression network. This method does not rely on a predefined anchor box, but directly outputs the coordinates of the center point of the target.

[0119] Center point prediction formula:

[0120] (x c ,y c )=Pred center (x c ,y c )

[0121] Among them, (x c ,y c ) is the predicted target center point coordinate, and Predcenter is the prediction function of the regression network for the center point position.

[0122] Step S2072, target size prediction. In addition to the center point, the Anchor-Free detection method also regresses and predicts the size (width and height) of the target. This step avoids the complexity of anchor box selection in traditional methods. Size prediction formula:

[0123] (u,h)=Predsize(@,h)

[0124] Where (w,h) is the predicted target size (width and height), and Predsize is the prediction function of the regression network for the target size.

[0125] Step S2073, target prediction process. The input image is preprocessed and then sent to the network for processing. The network directly outputs the center point and size of the target through regression prediction. This process uses convolutional layers and fully connected layers to perform regression calculations of position and size. Based on the predicted center point and size, the position and bounding box of the target are calculated. Finally, the detection result is generated.

[0126] The Anchor-Free detection method simplifies the detection process and reduces the complexity of anchor box selection by directly regressing the center point and size of the target. It improves the detection accuracy, especially in the case of complex backgrounds and multi-scale targets, and simplifies the design and training process of the model.

[0127] Step S208, improve the loss function. In order to improve the accuracy of the electric vehicle without helmet detection system, it is necessary to optimize the loss function to better meet the needs of the target detection task. The design of the loss function takes into account the position and size of the target to comprehensively evaluate the quality of the model prediction. The specific improvement process includes the following steps:

[0128] Step S2081, position loss is used to measure the error between the predicted target center point and the real target center point. The prediction of the target center point is the key to the Anchor-Free detection method. Position loss formula:

[0129]

[0130] Among them, (x c,i ,y c,i ) is the true center point coordinate of the i-th target, (x^ c,i ,y^ c,i ) is the predicted center point coordinate of the ith target, and N is the total number of targets.

[0131] The position loss is calculated as the sum of the squares of the Euclidean distances between the predicted center point and the true center point, reflecting the accuracy of the predicted position.

[0132] Step S2082, size loss is used to measure the difference between the predicted object size (width and height) and the actual object size. Size prediction is an essential part of object detection. Size loss formula:

[0133]

[0134] Among them, (w i ,h i ) is the true size (width and height) of the i-th object, (w^ i ,h^ i) is the predicted size (width and height) of the i-th object, and N is the total number of objects. The size loss calculates the squared difference between the predicted size and the true size, which measures the accuracy of the object size prediction.

[0135] Step S2083, comprehensive loss function. The comprehensive loss function combines the position loss and the size loss and weights them using a weight coefficient to achieve a balance between different tasks. Comprehensive loss function formula:

[0136] L total =α·L center +β·L size

[0137] Among them, α is the weight coefficient of position loss, β is the weight coefficient of size loss, Lcenter is the position loss, and Lsize is the size loss.

[0138] By introducing an improved loss function, the system can better balance the accuracy of position and size prediction, thereby improving the overall effect of electric vehicle helmet wearing detection. Comprehensive consideration of the center point and size of the target ensures the accuracy and reliability of the detection results, thereby enhancing the real-time monitoring capability of the system.

[0139] Preferably, the size and weight of the convolution kernel are dynamically adjusted in the convolution operation to adapt to different feature scales and data distributions. This method can be more flexible in dealing with objects of different scales, thereby improving detection accuracy.

[0140] In addition, super-resolution algorithms (such as SRCNN and ESRGAN) can be used in the preprocessing stage to enhance the low-resolution images so that the subsequent detection network can better process the details.

[0141] In addition, a multimodal data fusion network can be designed to input different types of data into the network and improve the detection performance through specific fusion strategies (such as splicing and weighted fusion).

[0142] According to the above steps, the embodiment of the present application has the following improvements and advantages:

[0143] 1. Using the improved ResNet structure (ResNetV2), the efficiency of feature extraction and the convergence speed of the model are improved by introducing batch normalization and optimized activation functions in the residual connection. Specifically, it improves batch normalization and ReLU activation functions in the residual block. It enhances the network's expressiveness, improves the training speed and convergence of the model, and makes the model more efficient in the feature extraction stage, which is suitable for application scenarios that require high-precision feature extraction.

[0144] 2. Mixed convolution (MixConv) is introduced to fuse convolution kernels of different sizes (such as 3×3, 5×5, and 7×7) to extract multi-scale features. Through convolution operations and feature fusion, a feature map with rich scale information is formed. This improves the diversity and comprehensiveness of feature extraction, enables the model to better identify objects of different scales, and improves detection accuracy and robustness.

[0145] 3. Use improved FPN (such as PANet) to further improve detection accuracy by enhancing the fusion capability between features at different levels. Multi-scale feature representation is enhanced through bottom-up and top-down feature fusion mechanisms, as well as feature fusion formulas. This improves detection accuracy and reduces false detections and missed detections. Improved FPN enhances the fusion between features at different levels, making the system more accurate in detecting targets at various scales.

[0146] 4. The cross-layer feature fusion module (ASFF) is introduced to further improve the detection capability of multi-scale targets by adaptively adjusting the weights of feature maps at different levels. Specifically, it applies adaptive weights to feature maps of different scales and optimizes them through feature fusion formulas. The weighting and fusion process of feature maps is optimized, the detection capability of multi-scale targets is improved, the model can better handle targets of various sizes, and the overall detection performance of the system is enhanced.

[0147] 5. Use improved Anchor-Free detection methods (such as CenterNet) to reduce the complexity of anchor box selection by predicting the center point and size of the target through regression. Including target center point prediction and target size prediction. Reduce the complexity of anchor box selection and improve the efficiency and accuracy of target detection. By directly predicting the center point and size of the target, the detection process is simplified, which is suitable for application scenarios that require high-efficiency target detection.

[0148] 6. A loss function that comprehensively considers the position and size of the target is designed, including position loss and size loss. By optimizing the loss function formula, the loss calculation is more comprehensive and accurate. The prediction accuracy of the model for the target position and size is improved, and the accuracy of target detection is further improved. The loss function that comprehensively considers the position and size optimizes the overall performance of target detection and enhances the detection reliability of the system.

[0149] Through the above improvements, the present invention realizes efficient real-time electric vehicle helmet wearing monitoring, and comprehensively considers multi-scale feature extraction, feature fusion, target detection and loss function optimization to ensure the accuracy and reliability of detection.

[0150] Figure 3 is a structural diagram of a model training device according to an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0151] The acquisition module 31 is used to acquire multiple image frames in the video stream, wherein each image frame includes: a target object wearing a target device.

[0152] The first processing module 32 is used to perform a convolution operation on the image frame, and perform batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map.

[0153] The second processing module 33 is used to perform a convolution operation on the first target feature map using convolution kernels of different scales, and fuse the output feature maps of the convolution kernels of different scales to obtain a second target feature map of the n-layer output of the convolution layer, where n is a positive integer greater than 1.

[0154] The fusion module 34 is used to perform multi-scale feature fusion on the second target feature map of each layer in the n layers of second target feature maps to obtain the first feature corresponding to the second target feature map of each layer, and perform weighted fusion on the first features to obtain a third target feature map.

[0155] The training module 35 is used to train the deep learning model using the third target feature map to obtain a target equipment recognition model, wherein the target equipment recognition model is used to recognize whether the target object is wearing the target equipment.

[0156] Optionally, the first processing module 32 is also used to perform the following steps: in each residual block of the backbone network, convolution operation is performed on the image frame, and the features obtained by the convolution operation are batch normalized to obtain a first feature map; in each residual block of the backbone network, convolution processing is performed on the first feature map using a 1×1 convolution kernel, and the features after convolution processing are extracted using an n×n convolution kernel, and convolution processing is performed on the extracted features using a 1×1 convolution kernel to obtain a second feature map, wherein n is a positive integer greater than 1; activation processing is performed on the second feature map using a preset activation function to obtain a third feature map; and the third feature map is added to the image frame to obtain a first target feature map.

[0157] Optionally, the fusion module 34 is also used to perform the following steps: for the second target feature map of the i-th layer, upsample the second target feature map of the i+1-th layer, and fuse the upsampled result with the second target feature map of the i-th layer to obtain a first fusion result, wherein i is a positive integer not greater than n; for the second target feature map of the i-th layer, downsample the second target feature map of the i-1-th layer, and fuse the downsampled result with the second target feature map of the i-th layer to obtain a second fusion result; for the second target feature map of the i-th layer, fuse the first fusion result and the second fusion result to obtain the first feature corresponding to the second target feature map of each layer.

[0158] Optionally, the fusion module 34 is also used to perform the following steps: determine the weight of each first feature based on the first parameter and the second parameter corresponding to each first feature, wherein the first parameter is used to characterize the expressive power of the first feature, and the second parameter is used to characterize the contribution of the first feature to the recognition task; and perform weighted fusion of the first features according to the weight of each first feature to obtain a third target feature map.

[0159] Optionally, the first parameter includes at least information entropy; the second parameter includes at least mutual information, wherein the mutual information is used to characterize the correlation between the feature map output by the deep learning model and the preset task label.

[0160] Optionally, the training module 35 is also used to perform the following steps: predict the coordinates of the center point of the target equipment in the third target feature map using the prediction function of the regression network for the center point; predict the size of the target equipment in the third target feature map using the prediction function of the size of the regression network; calculate the position and bounding box of the target equipment based on the predicted coordinates of the center point of the target equipment and the size of the target equipment to obtain the model output result; calculate the first error between the predicted coordinates of the center point of the target equipment and the coordinates of the actual center point, and calculate the second error between the predicted size of the target equipment and the actual size; perform weighted summation of the first error and the second error to obtain a loss function; and train the deep learning model using the third target feature map, and obtain a target equipment recognition model when the loss function meets preset requirements.

[0161] Optionally, the model training device is also used to perform the following steps before performing convolution operations on the image frames and performing batch normalization processing to obtain the first target feature map: using median filtering or Gaussian filtering to remove noise in the image frames; normalizing the pixel values ​​in the image frames to the range of [0, 1], and adjusting the size of the image frames to the target size required by the model input; randomly cropping the image frames, and rotating the image frames at random angles.

[0162] It should be noted that the above Figure 3 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0163] It should be noted that Figure 3 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0164] Figure 4FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a model training method. Figure 4 As shown, the computer terminal 40 may include one or more (402a, 402b, ..., 402n are used to illustrate) processors 402 (the processor 402 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 404 for storing data, and a transmission module 406 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0165] It should be noted that the one or more processors 402 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 40. As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0166] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in the embodiment of the present application. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, that is, the above-mentioned model training method is realized. The memory 404 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 404 may further include a memory remotely arranged relative to the processor 402, and these remote memories may be connected to the computer terminal 40 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The transmission module 406 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 40. In one example, the transmission module 406 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0168] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 40 .

[0169] It should be noted that, in some optional embodiments, the above Figure 4 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 4 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0170] It should be noted that Figure 4 The computer terminal shown is used to execute Figure 1 The model training method shown, therefore the relevant explanations in the execution method of the above commands are also applicable to the electronic device and will not be repeated here.

[0171] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above model training method.

[0172] A program for a non-volatile storage medium to perform the following functions: acquiring multiple image frames in a video stream, wherein each image frame includes: a target object wearing a target device; performing a convolution operation on the image frame, and performing batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map; performing a convolution operation on the first target feature map using convolution kernels of different scales, and fusing the output feature maps of the convolution kernels of different scales to obtain a second target feature map of n layers output by the convolution layer, wherein n is a positive integer greater than 1; performing multi-scale feature fusion on the second target feature map of each layer in the second target feature map of the n layers to obtain a first feature corresponding to the second target feature map of each layer, and performing weighted fusion on the first features to obtain a third target feature map; and training a deep learning model using the third target feature map to obtain a target device recognition model, wherein the target device recognition model is used to recognize whether the target object is wearing the target device.

[0173] An embodiment of the present application also provides an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above model training method is executed when the program is running.

[0174] The processor is used to run a program that performs the following functions: obtaining multiple image frames in a video stream, wherein each image frame includes: a target object wearing a target device; performing a convolution operation on the image frame, and performing batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map; performing a convolution operation on the first target feature map using convolution kernels of different scales, and fusing the output feature maps of the convolution kernels of different scales to obtain an n-layer second target feature map output by the convolution layer, wherein n is a positive integer greater than 1; performing multi-scale feature fusion on the second target feature map of each layer in the n-layer second target feature map to obtain a first feature corresponding to the second target feature map of each layer, and performing weighted fusion on the first features to obtain a third target feature map; and using the third target feature map to train a deep learning model to obtain a target device recognition model, wherein the target device recognition model is used to recognize whether the target object is wearing the target device.

[0175] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0176] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0177] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0182] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Acquire multiple image frames in a video stream, wherein each of the image frames includes: a target object wearing a target device; Performing a convolution operation on the image frame, and performing batch normalization processing on features obtained by the convolution operation to obtain a first target feature map; Using convolution kernels of different scales to perform a convolution operation on the first target feature map, and fusing the output feature maps of the convolution kernels of different scales to obtain a second target feature map of n layers output by the convolution layer, where n is a positive integer greater than 1; Perform multi-scale feature fusion on the second target feature map of each layer in the n layers of second target feature maps to obtain a first feature corresponding to the second target feature map of each layer, and perform weighted fusion on the first features to obtain a third target feature map; The deep learning model is trained using the third target feature map to obtain a target equipment recognition model, wherein the target equipment recognition model is used to identify whether the target object is wearing the target equipment.

2. The method according to claim 1, characterized in that: Performing a convolution operation on the image frame, and performing batch normalization processing on the features obtained by the convolution operation to obtain a first target feature map, including: In each residual block of the backbone network, a convolution operation is performed on the image frame, and features obtained by the convolution operation are batch normalized to obtain a first feature map; In each residual block of the backbone network, the first feature map is convolved using a 1×1 convolution kernel, the features after the convolution are extracted using an n×n convolution kernel, and the features after the extraction are convolved using a 1×1 convolution kernel to obtain a second feature map, where n is a positive integer greater than 1; Activating the second feature map using a preset activation function to obtain a third feature map; The third feature map is added to the image frame to obtain a first target feature map.

3. The method according to claim 1, characterized in that: Multi-scale feature fusion is performed on the second target feature map of each layer in the n layers of second target feature maps to obtain the first feature corresponding to the second target feature map of each layer, including: For the second target feature map of the i-th layer, up-sample the second target feature map of the i+1-th layer, and fuse the up-sampling result with the second target feature map of the i-th layer to obtain a first fusion result, where i is a positive integer not greater than n; For the second target feature map of the i-th layer, down-sample the second target feature map of the i-1-th layer, and fuse the down-sampling result with the second target feature map of the i-th layer to obtain a second fusion result; For the second target feature map of the i-th layer, the first fusion result and the second fusion result are fused to obtain the first feature corresponding to the second target feature map of each layer.

4. The method according to claim 1 or 3, characterized in that: The first features are weightedly fused to obtain a third target feature map, including: Determining a weight of each of the first features according to a first parameter and a second parameter corresponding to each of the first features, wherein the first parameter is used to characterize the expressiveness of the first feature, and the second parameter is used to characterize the contribution of the first feature to the recognition task; The first features are weightedly fused according to the weight of each of the first features to obtain the third target feature map.

5. The method according to claim 4, characterized in that The first parameter includes at least information entropy; the second parameter includes at least mutual information, wherein the mutual information is used to characterize the correlation between the feature map output by the deep learning model and the preset task label.

6. The method according to claim 1, characterized in that The deep learning model is trained using the third target feature map to obtain a target equipment recognition model, including: Predicting the coordinates of the center point of the target equipment in the third target feature map using a prediction function of the center point of the regression network; Predicting the size of the target equipment in the third target feature map using a prediction function of the size through a regression network; Calculate the position and bounding box of the target equipment according to the predicted coordinates of the center point of the target equipment and the size of the target equipment to obtain a model output result; Calculating a first error between the predicted coordinates of the center point of the target equipment and the actual coordinates of the center point, and calculating a second error between the predicted size of the target equipment and the actual size; Performing a weighted summation on the first error and the second error to obtain a loss function; The deep learning model is trained using the third target feature map, and when the loss function meets preset requirements, the target equipment recognition model is obtained.

7. The method according to claim 1, characterized in that Before performing a convolution operation and batch normalization processing on the image frame to obtain a first target feature map, the method further includes: Using median filtering or Gaussian filtering to remove noise in the image frame; Normalizing the pixel values ​​in the image frame to the range of [0, 1], and adjusting the size of the image frame to the target size required by the model input; The image frame is randomly cropped and the image frame is rotated at a random angle.

8. A method for identifying a target device, characterized in that: include: Acquire an image to be identified, wherein the image to be identified includes: a target object; A target equipment recognition model is used to identify whether the target object in the image to be recognized is wearing the target equipment, wherein the target equipment recognition model is trained by the model training method described in any one of claims 1 to 7.

9. A model training device, characterized in that: include: An acquisition module is used to acquire multiple image frames in a video stream, wherein each of the image frames includes: a target object wearing a target device; A first processing module, used for performing a convolution operation on the image frame, and performing batch normalization processing on features obtained by the convolution operation to obtain a first target feature map; A second processing module is used to perform a convolution operation on the first target feature map using convolution kernels of different scales, and fuse the output feature maps of the convolution kernels of different scales to obtain a second target feature map of n layers output by the convolution layer, where n is a positive integer greater than 1; A fusion module is used to perform multi-scale feature fusion on the second target feature map of each layer in the n layers of second target feature maps to obtain a first feature corresponding to the second target feature map of each layer, and perform weighted fusion on the first features to obtain a third target feature map; A training module is used to train a deep learning model using the third target feature map to obtain a target equipment recognition model, wherein the target equipment recognition model is used to identify whether the target object is wearing the target equipment.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the model training method described in any one of claims 1 to 7.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the model training method described in any one of claims 1 to 7 when running.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method described in any one of claims 1 to 7 is implemented.

Images