Millimeter wave image recognition model training method and device, and image recognition method and device

Abstract

The invention provides a training method of a millimeter wave image recognition model, and an image recognition method and device, and the method comprises the steps: adding a context extraction module into a backbone network of a recognition model, so as to obtain an initial recognition model; obtaining training data; for the millimeter wave image of any type of tool, determining a target image feature map corresponding to the millimeter wave image; and determining attention output corresponding to the millimeter wave image based on the target image feature map, obtaining multiple attention outputs corresponding to the millimeter wave images of different types of tools carried by the target object, adding the multiple attention outputs to the initial recognition model, and determining a trained recognition model. According to the technical scheme provided by the embodiment of the invention, the millimeter wave images corresponding to different types of tools are trained, the attention outputs corresponding to the millimeter wave images are obtained through calculation, the attention outputs corresponding to the different types of tools are obtained, and then the attention outputs are added into the initial recognition module, so that the training efficiency of the recognition model is greatly improved.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of image processing, and particularly relates to a training method of a millimeter wave image recognition model, an image recognition method and device. BACKGROUND

[0002] In the identification of dangerous goods in the security industry, security imaging recognition is mainly relied on, and the core is how to train a recognition model for identifying dangerous goods.

[0003] In the training of a traditional recognition model, manual feature extraction and shallow learning model training are mainly relied on. However, this training method can only be used for high-precision images, and for images with complex backgrounds and low imaging resolution, the training efficiency is low, and the recognition accuracy of the trained model is not high. SUMMARY

[0004] Embodiments of the present disclosure provide a solution to solve the problem of low training efficiency and low recognition accuracy of the trained model in the related art.

[0005] In a first aspect, the present disclosure provides a training method of a millimeter wave image recognition model, comprising:

[0006] adding a context extraction module to a backbone network in the recognition model to obtain an initial recognition model;

[0007] obtaining training data, wherein the training data comprises millimeter wave images of a target object carrying different types of tools;

[0008] for a millimeter wave image of any type of tool, determining a target image feature map corresponding to the millimeter wave image;

[0009] based on the target image feature map, determining an attention output corresponding to the millimeter wave image, to obtain a plurality of attention outputs corresponding to the millimeter wave images of the target object carrying different types of tools, wherein the attention output includes a channel attention output and a spatial attention output;

[0010] adding the plurality of attention outputs to the initial recognition model to determine a trained recognition model.

[0011] In a second aspect, the present disclosure provides an image recognition method, comprising:

[0012] obtaining a millimeter wave image to be recognized;

[0013] inputting the millimeter wave image to be recognized into the trained recognition model to obtain a recognition result of a tool in the millimeter wave image to be recognized; the recognition result at least includes: category information, spatial position information;

[0014] The trained recognition module is obtained by the training method of the millimeter wave image recognition model according to any one of claims 1 to 7.

[0015] In a third aspect, the present disclosure provides a device for training a millimeter wave image recognition model, the device comprising:

[0016] A first obtaining unit is configured to add the context extraction module to a backbone network in the recognition model to obtain an initial recognition model.

[0017] The first obtaining unit is further configured to obtain training data, the training data comprising millimeter wave images of a target object carrying different types of tools.

[0018] A determining unit is configured to determine, for a millimeter wave image of any type of tool, a target image feature map corresponding to the millimeter wave image.

[0019] The determining unit is further configured to determine, based on the target image feature map, an attention output corresponding to the millimeter wave image, to obtain a plurality of attention outputs corresponding to the millimeter wave images of the target object carrying different types of tools, wherein the attention output comprises a channel attention output and a spatial attention output.

[0020] The determining unit is further configured to add the plurality of attention outputs to the initial recognition model to determine a trained recognition model.

[0021] In a fourth aspect, the present disclosure provides an image recognition device, the device comprising:

[0022] A second obtaining unit is configured to obtain a millimeter wave image to be recognized.

[0023] A recognition unit is configured to input the millimeter wave image to be recognized into the trained recognition model to obtain a recognition result of a tool in the millimeter wave image to be recognized, the recognition result comprising at least: category information, spatial position information.

[0024] The trained recognition module is obtained by the training method of the millimeter wave image recognition model according to the first aspect.

[0025] The technical solution provided in this disclosure adds a context extraction module to the backbone network of the recognition model to obtain an initial recognition model; acquires training data, which includes millimeter-wave images of a target object carrying different types of tools; for any type of tool, determines the target image feature map corresponding to the millimeter-wave image; based on the target image feature map, determines the attention output corresponding to the millimeter-wave image, obtaining multiple attention outputs corresponding to millimeter-wave images of the target object carrying different types of tools, wherein the attention outputs include channel attention outputs and spatial attention outputs; and adds the multiple attention outputs to the initial recognition model to determine the trained recognition model. The technical solutions provided in the various embodiments of this disclosure, by training on millimeter-wave images corresponding to different types of tools, calculate their corresponding attention outputs, and obtain the attention outputs corresponding to different types of tools, and then adding the attention outputs to the initial recognition module, greatly improve the training efficiency of the recognition model. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0027] Figure 1 A flowchart illustrating a training method for a millimeter-wave image recognition model provided in an embodiment of this disclosure;

[0028] Figure 2 This is a schematic diagram illustrating different body postures of a target object according to an embodiment of the present disclosure;

[0029] Figure 3 A schematic flowchart of an image recognition method provided in an embodiment of this disclosure;

[0030] Figure 4 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present disclosure;

[0031] Figure 5 A schematic diagram of the structure of a training device for a millimeter-wave image recognition model provided in an embodiment of this disclosure;

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. Detailed Implementation

[0033] Embodiments of the present disclosure are described below in detail, examples of which are shown in the accompanying drawings. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present disclosure, and cannot be understood as a limitation of the present disclosure.

[0034] The terms "first" and "second" and the like in the specification of the embodiments of the present disclosure, claims, and accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a particular order or chronological order. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented, for example, in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or apparatuses.

[0035] The training method of the millimeter wave image recognition model provided by the embodiments of the present disclosure can run on a terminal device or a server. The terminal device can be a local terminal device, including wearable devices such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0036] In the security industry, the identification of dangerous goods mainly relies on security imaging identification, and the most core is how to train an identification model for identifying dangerous goods.

[0037] In the traditional identification model training, manual feature extraction and shallow learning model training are mainly relied on. However, this training method can only be used for high-precision images, and for images with complex background and low imaging resolution, the training efficiency is low, and the recognition accuracy of the trained model is not high.

[0038] Figure 1 A flowchart of a training method of a millimeter wave image recognition model is provided for an exemplary embodiment of the present disclosure, which includes at least the following steps S101-S105:

[0039] S101, a context extraction module is added to a backbone network in the identification model to obtain an initial identification model.

[0040] In some embodiments, the recognition model is preferably a recognition model based on a transformer architecture, consisting of a backbone network, a Transformer encoder and a decoder with an auxiliary prediction head.

[0041] Further, the initial recognition model is a recognition model formed by adding a context extraction module with context information extraction function to the backbone network of the recognition model based on the transformer architecture.

[0042] S102, obtaining training data.

[0043] In some embodiments, the training data includes millimeter wave images of target objects carrying different types of tools. Wherein, the target object is preferably a passenger, and the different types of tools are preferably different types of dangerous goods, such as knives, scissors and other articles.

[0044] Specifically, the training data mainly refers to data collected in a non-cooperative security scene, i.e. the target object is in motion when collecting data. For example, the target object (passenger) carries different sizes of knives, scissors, hammers and other dangerous goods, and collects data when raising hands, squatting, spreading arms or walking.

[0045] In some embodiments, in the actual acquisition process, Figure 2 As shown, the volunteers (target objects) synchronously collect millimeter wave echo data and RGB images after making different body postures, and then obtain corresponding millimeter wave images. Wherein, the number of millimeter wave images obtained is not limited, and the conversion of millimeter wave echo data to millimeter wave images is not limited.

[0046] In some embodiments, the millimeter wave image refers to the current frame image of the target object carrying the tool.

[0047] S103, determining the target image feature map corresponding to the millimeter wave image of any type of tool.

[0048] In some embodiments, the determination of the target image feature map corresponding to the millimeter wave image includes steps S11-S14:

[0049] S11, obtaining the first detailed feature and the second detailed feature of the tool in the millimeter wave image through the backbone network in the initial target recognition model.

[0050] In the actual recognition process, it is necessary to recognize the tool contained in the millimeter wave image, so it is necessary to recognize the first detailed feature and the second detailed feature of the tool contained in the millimeter wave image, wherein the first detailed feature contains the edge feature and the texture feature of the tool, and the second detailed feature includes the semantic feature of the tool.

[0051] Specifically, the backbone network in the initial target recognition model extracts edge features and texture features of the tool in the millimeter wave image at a low level, for positioning the position of the tool.

[0052] Specifically, the backbone network in the initial target recognition model captures semantic features of the tool in the millimeter wave image at a high level, for identifying the type of the tool.

[0053] S12, obtaining a first image feature map based on the first detailed feature and the second detailed feature.

[0054] In some embodiments, the obtaining of the first image feature map based on the first detailed feature and the second detailed feature comprises fusing the first detailed feature and the second detailed feature to obtain the first image feature map. The fusion manner is not limited herein.

[0055] S13, obtaining a third detailed feature of the tool in the millimeter wave image through the context extraction module based on the first image feature map.

[0056] In some embodiments, the third detailed feature comprises a local context feature and a surrounding context feature of the tool.

[0057] In the actual obtaining process, the local context feature is mainly obtained through convolution operation, and the surrounding context feature is mainly obtained through dilated convolution. The specific obtaining formula is as follows:

[0058] The local context feature is: F loc = ChannelWiseConv(X, W loc );

[0059] The surrounding context feature is: Fs sur = ChannelWiseDilatedConv(X, W sur , dilation);

[0060] Wherein, the X is the first image feature map, the W loc is the convolution kernel of the local context feature, and the W sur is the convolution kernel of the surrounding context feature.

[0061] S14, determining the target image feature map based on the third detailed feature.

[0062] In some embodiments, the determining of the target image feature map based on the third detailed feature comprises steps S141-S143:

[0063] S141, obtain a joint feature based on the local context feature and the surrounding context feature.

[0064] In some actual recognition processes, the local context feature and the surrounding context feature need to be spliced, and then batch normalization and ReLU activation function processing are performed to obtain a joint feature. The specific calculation formula is as follows:

[0065] The joint feature is:

[0066] The BN is a normalization operation, and the is splicing features according to channels.

[0067] S142, obtain a global context feature corresponding to the millimeter wave image based on the joint feature through global pooling.

[0068] In some embodiments, the global context feature is:

[0069] F glo =F joi ×σ(W2×ReLU(W1×GlobalAvgPool(F joi ))

[0070] The × is element-wise multiplication, the W2 and the W1 are weight matrices of a full connection layer, and the σ is a Sigmoid activation function.

[0071] In actual operation, introducing the global context feature of the target object (passenger) can help the recognition model to judge the influence of the human body posture on dangerous goods, consider the background information of the entire image, and reduce misjudgment caused by local shielding or background complexity.

[0072] S143, obtain the target image feature map based on the global context feature and the joint feature.

[0073] In some other embodiments, without introducing the global context feature of the human body, determining the target image feature map based on the third detail feature includes: fusing the local context feature and the surrounding context feature to obtain the target image feature map.

[0074] S104, determine the attention output corresponding to the millimeter wave image based on the target image feature map, and obtain multiple attention outputs corresponding to the millimeter wave image of the target object carrying different types of tools.

[0075] The attention output includes a channel attention output and a spatial attention output.

[0076] In some embodiments, in actual operation, in order to further enhance the perception of the occluded area (the area of the occlusion tool), a spatial attention mechanism needs to be adopted to optimize the target image feature map and the deformation feature jointly to improve the modeling capability of the network for local and occluded areas, and the specific process is as follows:

[0077] Based on the target image feature map, the attention output corresponding to the millimeter wave image is determined, including steps S21-S22:

[0078] S21, based on the target image feature map, a deformation feature is obtained through deformable convolution.

[0079] Wherein, the deformation feature is: O = DeformConvd (X, Delta, padding = 1), the X is the target image feature map, and the Delta is the offset.

[0080] In this embodiment, the offset can help the model adjust the sampling position of the convolution kernel in the occluded area and improve the information loss caused by the occlusion.

[0081] S22, based on the deformation feature and the target image feature map, the spatial attention output is calculated.

[0082] In some embodiments, the spatial attention output is: S attn = sigma (W s x [X, O]).

[0083] Wherein, the W s is a 1x1 convolution kernel, and the [X, O] is a splicing of the target image feature map and the deformation feature.

[0084] In this embodiment, through this process, the spatial information of the occluded area can be effectively retained and weighted, which is beneficial to the training of the model.

[0085] In some embodiments, in the actual training process, there will be an occlusion problem in the image. Because the inter-channel dependency relationship of the occlusion problem is affected, the local channel information modeling of the recognition model is insufficient, and then the grouping cross-channel attention mechanism is needed to calculate the self-attention in multiple attention heads in parallel to promote the information interaction across the heads, thereby enhancing the relevance and feature expression between channels. Even in the case of occlusion, each head can independently capture the feature information of the occluded area. Therefore, the introduced channel attention output is as follows:

[0086] Based on the target image feature map, the attention output corresponding to the millimeter wave image is determined, including: based on the spatial attention output and the channel attention weight matrix, the channel attention output is obtained.

[0087] The channel attention output is C' attn = Proj(C attn ) ;

[0088] wherein C attn = A x V; Q, K, V = Chunk(Conv(X), 3); Q, K, and V are query, key, and value of the target image feature map, d is the dimension of each head, and A is a channel attention weight matrix.

[0089] S105, adding multiple attention outputs to the initial recognition model to determine a trained recognition model.

[0090] In some embodiments, in order to further enhance the synergy of spatial and channel attention, it is necessary to weight the spatial attention output and the channel attention output. Specifically, the spatial attention output and the channel attention output are fused by generating fusion weights through 1x1 convolution, which adaptively adjusts the attention degree to different regions, enhances or suppresses the influence of the occluded region. The specific process is as follows:

[0091] The adding of the multiple attention outputs to the initial recognition model to determine the trained recognition model comprises: weighting the spatial attention output and the channel attention output contained in each attention output by using a dynamic fusion mechanism to obtain multiple weighted multiple attention outputs, and adding the multiple weighted attention outputs to the initial recognition model to determine the trained recognition model.

[0092] wherein the weighted attention output is O fusion = X x (W fusion [0] x S attn + W fusion [1] x C' attn ) ;

[0093] The W fusion is generated by 1x1 convolution and normalized by using Softmax.

[0094] In some embodiments, in order to ensure that the obtained trained recognition model has a very high recognition rate, we need to verify the trained recognition model, and the specific steps are as follows steps S31-S33:

[0095] S31, obtaining sample data containing multiple millimeter wave images of the target object carrying the same type of tool and related information of the corresponding tool, wherein the related information includes category information and spatial position information of the tool.

[0096] S32, identifying each type of millimeter wave image contained in the sample data through the trained recognition model to obtain different recognition results.

[0097] S33, comparing different recognition results with corresponding relevant information to determine the recognition rate of the trained recognition model.

[0098] In some embodiments, the number of millimeter wave images contained in the sample data obtained in step S31 is not limited.

[0099] Further, the plurality of millimeter wave images can be images of the target object carrying the same type of tool in different body postures.

[0100] By identifying the millimeter wave images in different postures, the recognition accuracy of the trained recognition model for this type of tool can be obtained. When the accuracy is greater than the preset threshold, it can be determined that the recognition of this type of tool by the trained recognition model meets the requirements. Then, different types of tools are identified according to the above method. If the recognition accuracy of a certain type of tool is lower than the preset threshold, the trained recognition model needs to be retrained.

[0101] The technical solution provided by the present disclosure adds a context extraction module to the backbone network in the recognition model to obtain an initial recognition model; obtains training data, the training data including: millimeter wave images of the target object carrying different types of tools; for any type of tool, determines the target image feature map corresponding to the millimeter wave image; based on the target image feature map, determines the attention output corresponding to the millimeter wave image, to obtain a plurality of attention outputs corresponding to the millimeter wave images of the target object carrying different types of tools, wherein the attention output includes channel attention output and spatial attention output; adds the plurality of attention outputs to the initial recognition model to determine the trained recognition model. The technical solution provided by each embodiment of the present disclosure trains the millimeter wave images corresponding to different types of tools, calculates the corresponding attention output, obtains the attention output corresponding to different types of tools, and then adds the attention output to the initial recognition model, thereby greatly improving the training efficiency of the recognition model.

[0102] Figure 3 A flowchart of an image recognition method provided by an exemplary embodiment of the present disclosure is suitable for an inspection device, and the scheme includes at least the following steps S201-S202:

[0103] S201, obtaining a millimeter wave image to be identified.

[0104] In some embodiments, the millimeter wave image to be identified refers to a millimeter wave image of a target object.

[0105] Specifically, taking the subway security check scene as an example, the millimeter wave image of the target object refers to the millimeter wave image collected at the current moment of the passenger. Wherein, the passenger is in motion.

[0106] S202, input the millimeter wave image to be identified into the trained identification model to obtain the identification result of the tool in the millimeter wave image to be identified.

[0107] In some embodiments, the identification result at least includes: category information, spatial position information.

[0108] Wherein, the trained identification model is obtained by the above-mentioned millimeter wave image identification model training method.

[0109] In some embodiments, inputting the millimeter wave image to be identified into the trained identification model to obtain the identification result of the tool in the millimeter wave image to be identified includes steps S41-S44:

[0110] S41, the identification model determines the second detailed feature of the tool in the millimeter wave image to be identified in response to the millimeter wave image to be identified as input.

[0111] S42, determine the type of tool based on the second detailed feature.

[0112] S43, based on the type of tool, determine the target attention output corresponding to the type of tool from multiple attention outputs.

[0113] S44, the trained identification model calls the target attention output to identify the millimeter wave image to be identified to obtain the identification result.

[0114] Specifically, taking the subway security check scene as an example, if a passenger appears in the preset area, the inspection device will obtain the millimeter wave image of the passenger at the current moment, and input the millimeter wave image into the identification model, and extract the second detailed feature of the tool in the millimeter wave image. According to the second detailed feature, the type of the tool can be determined, and then the corresponding target attention output can be determined according to the type of the tool. Finally, the identification model calls the target attention model to identify the tool in the millimeter wave image, and then obtains the category information and spatial position information of the tool. By using this identification method, the identification accuracy of the identification model can be improved, and the identification efficiency is improved.

[0115] The technical scheme provided by the present disclosure obtains the millimeter wave image to be identified, inputs the millimeter wave image to be identified into the trained identification model, and obtains the identification result of the tool in the millimeter wave image to be identified. By using this identification method, the identification accuracy of the identification model can be improved, and the identification efficiency is improved.

[0116] Figure 4 A structural schematic diagram of a training device of a millimeter wave image recognition model is provided for an exemplary embodiment of the present disclosure;

[0117] The device comprises a first acquisition unit 301 and a determination unit 302.

[0118] The first acquisition unit 301 is configured to add a context extraction module to a backbone network in a recognition model to obtain an initial recognition model.

[0119] The first acquisition unit 301 is further configured to acquire training data, wherein the training data comprises millimeter wave images of a target object carrying different types of tools.

[0120] The determination unit 302 is configured to determine a target image feature map corresponding to a millimeter wave image of any type of tool.

[0121] The determination unit 302 is further configured to determine an attention output corresponding to the millimeter wave image based on the target image feature map, to obtain a plurality of attention outputs corresponding to the millimeter wave images of the target object carrying different types of tools, wherein the attention output comprises a channel attention output and a spatial attention output.

[0122] The determination unit 302 is further configured to add the plurality of attention outputs to the initial recognition model to determine a trained recognition model.

[0123] In some embodiments, the device is configured to determine the target image feature map corresponding to the millimeter wave image, and the device is specifically configured to:

[0124] Obtain a first detailed feature and a second detailed feature of a tool in the millimeter wave image through a backbone network in the initial recognition model, wherein the first detailed feature comprises an edge feature and a texture feature of the tool, and the second detailed feature comprises a semantic feature of the tool.

[0125] Obtain a first image feature map based on the first detailed feature and the second detailed feature.

[0126] Obtain a third detailed feature of the tool in the millimeter wave image through the context extraction module based on the first image feature map, wherein the third detailed feature comprises a local context feature and a surrounding context feature of the tool.

[0127] Determine the target image feature map based on the third detailed feature.

[0128] The local context feature is F locChannelWiseConv(X, W loc ); the surrounding context feature is F sur =ChannelWiseDilatedConv(X, W sur , dilation);

[0129] wherein, the X is the first image feature map, the W loc is a convolution kernel of the local context feature, and the W sur is a convolution kernel of the surrounding context feature.

[0130] In some embodiments, the device is configured to determine the target image feature map based on the third detail feature, and specifically configured to:

[0131] fuse the local context feature and the surrounding context feature to obtain the target image feature map.

[0132] In some embodiments, the device is configured to determine the target image feature map based on the third detail feature, and specifically configured to:

[0133] obtain a joint feature based on the local context feature and the surrounding context feature; the joint feature is: wherein, the BN is a normalization operation, and the is to concatenate features by channel;

[0134] obtain a global up-down feature corresponding to the millimeter wave image by global pooling based on the joint feature; the global up-down feature is: glo =F joi ×σ(W2×ReLU(W1×GlobalAvgPool(F joi ))); the × is element-wise multiplication, the W2 and the W1 are weight matrices of a full connection layer, and the σ is a Sigmoid activation function;

[0135] obtain the target image feature map based on the global up-down feature and the joint feature.

[0136] In some embodiments, the device is configured to determine the attention output corresponding to the millimeter wave image based on the target image feature map, and specifically configured to:

[0137] obtain a deformation feature by deformable convolution based on the target image feature map; the deformation feature is O=DeformConvd(X, Δ, padding=1), the X is a target image feature map, and the Δ is an offset;

[0138] Based on the deformation feature and the target image feature map, a spatial attention output is calculated; the spatial attention output is S attn = σ(W s × [X, O]), W s is a 1x1 convolution kernel, and [X, O] is a splicing of the target image feature map and the deformation feature.

[0139] In some embodiments, the device is configured to determine an attention output corresponding to the millimeter wave image based on the target image feature map, and specifically configured to:

[0140] Based on the spatial attention output and a channel attention weight matrix, a channel attention output is obtained.

[0141] The channel attention output is C' attn = Proj(C attn );

[0142] wherein C attn = A x V; Q, K, V = Chunk(Conv(X), 3); Q, K, and V are a query, a key, and a value of the target image feature map, respectively, d is a dimension of each head, and A is a channel attention weight matrix.

[0143] In some embodiments, the device is configured to add multiple attention outputs to an initial recognition model to determine a trained recognition model, and specifically configured to:

[0144] The dynamic fusion mechanism is used to weight the outputs of the spatial attention mechanism and the channel attention output contained in each attention output, to obtain multiple weighted multiple attention outputs, and the multiple weighted attention outputs are added to the initial recognition model to determine the trained recognition model.

[0145] The weighted attention output is O fusion = X x (W fusion [0] x S attn + W fusion [1] x C' attn );

[0146] W fusion is generated by 1x1 convolution and normalized by Softmax.

[0147] The technical scheme provided by the present disclosure adds a context extraction module to a backbone network in a recognition model to obtain an initial recognition model; training data is obtained, and the training data includes: millimeter wave images of a target object carrying different types of tools; for a millimeter wave image of any type of tool, a target image feature map corresponding to the millimeter wave image is determined; based on the target image feature map, an attention output corresponding to the millimeter wave image is determined, and a plurality of attention outputs corresponding to the millimeter wave images of the target object carrying different types of tools are obtained, wherein the attention output includes a channel attention output and a spatial attention output; the plurality of attention outputs are added to the initial recognition model to determine a trained recognition model. The technical scheme provided by each embodiment of the present disclosure trains the millimeter wave images corresponding to different types of tools, calculates the corresponding attention outputs, obtains the attention outputs corresponding to different types of tools, and then adds the attention outputs to the initial recognition model, thereby greatly improving the training efficiency of the recognition model.

[0148] Figure 5 A structural schematic diagram of an image recognition device provided for an exemplary embodiment of the present disclosure is shown in FIG. 1.

[0149] The device includes a second acquisition unit 401 and a recognition unit 402.

[0150] The second acquisition unit 401 is configured to acquire a millimeter wave image to be recognized.

[0151] The recognition unit 402 is configured to input the millimeter wave image to be recognized into a trained recognition model to obtain a recognition result of a tool in the millimeter wave image to be recognized.

[0152] In some embodiments, the recognition result at least includes: category information, spatial position information.

[0153] The trained recognition model is obtained by the training method of the millimeter wave image recognition model described in the above embodiments.

[0154] In some embodiments, the device is configured to input the millimeter wave image to be recognized into a trained recognition model to obtain a recognition result of a tool in the millimeter wave image to be recognized, and the device is specifically configured to:

[0155] The recognition model determines a second detailed feature of the tool in the millimeter wave image to be recognized in response to the millimeter wave image to be recognized as input.

[0156] Based on the second detailed feature, the type of the tool is determined.

[0157] determine, based on the type of the tool, a target attention output corresponding to the tool from a plurality of attention outputs;

[0158] The trained recognition model calls the target attention output, recognizes the to-be-recognized millimeter wave image, and obtains the recognition result.

[0159] The technical solution provided by the present disclosure obtains a to-be-recognized millimeter wave image, inputs the to-be-recognized millimeter wave image into a trained recognition model, and obtains a recognition result of a tool in the to-be-recognized millimeter wave image. By using this recognition method, the recognition accuracy of the recognition model can be improved, and the recognition efficiency is also improved.

[0160] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, details are not described here. Specifically, the device can perform the above-mentioned method embodiments, and the foregoing and other operations and / or functions of each module in the device are respectively for the corresponding processes in each method in the above-mentioned method embodiments, and for the sake of brevity, details are not described here.

[0161] The device of the embodiment of the present disclosure is described above in conjunction with the functional modules. It should be understood that the functional modules can be realized by hardware, or by instructions in the form of software, or by a combination of hardware and software modules. Specifically, each step of the method embodiments in the present disclosure can be completed by the integrated logic circuit of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in conjunction with the present embodiment can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned method embodiments in conjunction with its hardware.

[0162] Figure 6 is a schematic block diagram of an electronic device provided by the present embodiment, which can include:

[0163] The memory 501 is used to store computer programs and transmit the program codes to the processor 502. In other words, the processor 502 can call and run the computer program from the memory 501 to implement the method in the present embodiment.

[0164] For example, the processor 502 can be used to execute the above-mentioned method embodiments according to the instructions in the computer program.

[0165] In some embodiments of the present disclosure, the processor 502 can include, but is not limited to:

[0166] A general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, and the like.

[0167] In some embodiments of the present disclosure, the memory 501 includes, but is not limited to:

[0168] volatile memory and / or non-volatile memory. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synch link DRAM (SLDRAM), and a direct Rambus RAM (DR RAM).

[0169] In some embodiments of the present disclosure, the computer program can be divided into one or more modules, which are stored in the memory 501 and executed by the processor 502 to complete the method provided by the present disclosure. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0170] AsFigure 6 As shown, the electronic device can further include:

[0171] a transceiver 503, which can be connected to the processor 502 or the memory 501.

[0172] The processor 502 can control the transceiver 503 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 503 can include a transmitter and a receiver. The transceiver 503 can further include an antenna, and the number of antennas can be one or more.

[0173] It should be understood that various components in the electronic device are connected through a bus system, wherein the bus system includes a data bus, a power supply bus, a control bus, and a state signal bus in addition to a data bus.

[0174] The present disclosure also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above method embodiments. Alternatively, the present disclosure embodiments also provide a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above method embodiments.

[0175] When implemented by using software, the computer program product can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the flow or function according to the embodiments of the present disclosure in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, digital video disc (DVD)), or a semiconductor medium (for example, solid state disk (SSD)), etc.

[0176] Those skilled in the art can understand that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0177] In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0178] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e. can be located in one place or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. For example, the functional modules in each embodiment of the present disclosure can be integrated in one processing module, or each module can be physically present separately, or two or more modules can be integrated in one module.

[0179] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A training method for a millimeter-wave image recognition model, characterized in that, The method includes: The context extraction module is added to the backbone network of the recognition model to obtain the initial recognition model; Acquire training data, which includes millimeter-wave images of the target object carrying different types of tools; For millimeter-wave images of any type of tool, determine the target image feature map corresponding to the millimeter-wave image; Based on the target image feature map, the attention output corresponding to the millimeter-wave image is determined, and multiple attention outputs corresponding to millimeter-wave images of the target object carrying different types of tools are obtained. The attention outputs include channel attention outputs and spatial attention outputs. Multiple attention outputs are added to the initial recognition model to determine the trained recognition model.

2. The method according to claim 1, characterized in that, Determining the target image feature map corresponding to the millimeter-wave image includes: The first detail feature and the second detail feature of the tool in the millimeter-wave image are obtained through the backbone network in the initial recognition model. The first detail feature includes the edge feature and texture feature of the tool, and the second detail feature includes the semantic feature of the tool. Based on the first detail feature and the second detail feature, a first image feature map is obtained; Based on the first image feature map, the third detail feature of the tool in the millimeter-wave image is obtained through the context extraction module. The third detail feature includes the local context feature of the tool and the surrounding context feature. Based on the third detailed feature, the feature map of the target image is determined; The local context feature is: F loc =ChannelWiseConv(X, W) loc The surrounding context feature is: F sur =ChannelWiseDilatedConv(X,W sur (dilation); Wherein, X is the first image feature map, and W loc The convolution kernel for the local context features, the W sur The convolution kernel is the feature of the surrounding context.

3. The method according to claim 2, characterized in that, Based on the third detailed feature, the target image feature map is determined, including: The target image feature map is obtained by fusing the local context features and the surrounding context features.

4. The method according to claim 2, characterized in that, Based on the third detailed feature, the target image feature map is determined, including: Based on the local context features and the surrounding context features, a joint feature is obtained; the joint feature is: Wherein, BN is a normalization operation, and the To stitch features together according to channels; Based on the joint features, global pooling is used to obtain the global top and bottom features corresponding to the millimeter-wave image; the global top and bottom features are: F glo =F joi ×σ(W2×ReLU(W1×GlobalAvgPool(F joi ))); The × represents element-wise multiplication, W2 and W1 are the weight matrices of the fully connected layer, and σ is the Sigmoid activation function; Based on the global top and bottom features and the joint features, the target image feature map is obtained.

5. The method according to claim 3, characterized in that, Based on the target image feature map, determine the attention output corresponding to the millimeter-wave image, including: Based on the target image feature map, deformable features are obtained through deformable convolution; the deformable features are: O = DeformConvd(X, Δ, padding = 1), where X is the target image feature map and Δ is the offset; Based on the deformation features and the target image feature map, the spatial attention output is calculated; the spatial attention output is: S attn =σ(W s ×[X, O]), the W s The kernel is 1×1, and [X, O] is the concatenation of the target image feature map and the deformed feature.

6. The method according to claim 4, characterized in that, Based on the target image feature map, determine the attention output corresponding to the millimeter-wave image, including: Based on the spatial attention output and the channel attention weight matrix, the channel attention output is obtained; The channel attention output is: C′ attn =Proj(C attn ); in, Q, K, V = Chunk(Conv(X), 3); where Q, K, V are the query, key, and value of the target image feature map, respectively, d is the dimension of each head, and A is the channel attention weight matrix.

7. The method according to claim 1, characterized in that, Multiple attention outputs are added to the initial recognition model to determine the trained recognition model, including: A dynamic fusion mechanism is used to weight the spatial attention mechanism and channel attention output contained in each attention output, resulting in multiple weighted attention outputs. These multiple weighted attention outputs are then added to the initial recognition model to determine the trained recognition model. The weighted attention output is: O fusion =X×(W fusion [0]·S attn +W fision [1]·C′ attn ); The W fusion The result is generated by 1×1 convolution and then normalized using Sonftmax.

8. An image recognition method, characterized in that, The method includes: Acquire the millimeter-wave image to be identified; The millimeter-wave image to be identified is input into the trained recognition model to obtain the recognition result of the tool in the millimeter-wave image to be identified; the recognition result includes at least: category information and spatial location information; The trained recognition module is obtained by the training method of the millimeter-wave image recognition model according to any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, The millimeter-wave image to be identified is input into the trained recognition model to obtain the recognition result of the tool in the millimeter-wave image, including: The recognition model, in response to the millimeter-wave image to be recognized as input, determines the second detail features of the tool in the millimeter-wave image to be recognized; Based on the second detailed feature, the type of tool is determined; Based on the type of the tool, determine the target attention output corresponding to that type of tool from multiple attention outputs; The trained recognition model invokes the target attention output to recognize the millimeter-wave image to be recognized, and obtains the recognition result.

10. A training device for a millimeter-wave image recognition model, characterized in that, The device includes: The first acquisition unit is used to add the context extraction module to the backbone network of the recognition model in order to obtain the initial recognition model; The first acquisition unit is further configured to acquire training data, the training data including: millimeter-wave images of the target object carrying different types of tools; The determining unit is used to determine the target image feature map corresponding to the millimeter-wave image for any type of tool. The determining unit is further configured to determine the attention output corresponding to the millimeter-wave image based on the feature map of the target image, and obtain multiple attention outputs corresponding to the millimeter-wave images of the target object carrying different types of tools, wherein the attention output includes channel attention output and spatial attention output; The determining unit is also used to add multiple attention outputs to the initial recognition model to determine the trained recognition model.

11. An image recognition device, characterized in that, The device includes: The second acquisition unit is used to acquire the millimeter-wave image to be identified; The recognition unit is used to input the millimeter-wave image to be recognized into the trained recognition model to obtain the recognition result of the tool in the millimeter-wave image to be recognized; the recognition result includes at least: category information and spatial location information; The trained recognition module is obtained by the training method of the millimeter-wave image recognition model according to any one of claims 1 to 7.

Images