Model training method and device, target recognition method, equipment, medium

By using the dual-stream network model to extract and fusion the infrared and visible light images in target recognition, the problem of low accuracy of target recognition is solved, and a more efficient target recognition effect is achieved.

CN114937189BActive Publication Date: 2024-11-19PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210509810.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-11
Publication Date
2024-11-19
Estimated Expiration
2042-05-11

AI Technical Summary

Technical Problem

In the prior art, the recognition accuracy of target recognition is not high, especially in target re-identification, and it is difficult to effectively identify the same target in the preset image and the image dataset.

Method used

A model training method is proposed. By obtaining the original infrared image and visible light image, input it into the dual-stream network model for preliminary feature extraction and key feature extraction, combined with cross-modal fusion processing, the target recognition model is trained to improve recognition accuracy.

Benefits of technology

By extracting and fusing the features of infrared and visible images, the accuracy of target recognition is significantly improved, the features in the target image can be effectively identified, and the accuracy of the recognition results of the model is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114937189B_ABST
    Figure CN114937189B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a model training method and device, a target recognition method, equipment, and medium, which belong to the field of artificial intelligence technology. The model training method extracts features from the original visible light image to obtain a first feature image, extracts features from the original infrared image to obtain a second feature image, and then performs cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image, and then trains a dual-stream network model based on the first feature image, the second feature image, the first fused image, and the second fused image to obtain a target recognition model, which is used to perform recognition processing on the target image to obtain a target recognition result. The technical solution of the embodiments of the present application improves the accuracy of the model in recognizing the target image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method and device, a target recognition method, equipment, and medium. Background Art

[0002] Object recognition is usually used to identify related objects (such as pedestrians, vehicles, faces, etc.), such as object re-identification, where the general principle of object re-identification is: given a preset image including a target, find an image of the same target as the preset image from the image data set; in related technologies, the recognition accuracy of object re-identification is not high. Therefore, how to improve the accuracy of object recognition has become a technical problem that people skilled in the art need to solve urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method and device, target recognition method, equipment, and medium, aiming to improve the accuracy of the model in recognizing the target image.

[0004] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a model training method, the model training method comprising:

[0005] Acquire an original training image; wherein the original training image includes an original infrared image and an original visible light image, the original infrared image is obtained by photographing a preset target, and the original visible light image is obtained by photographing the preset target;

[0006] Inputting the original training image into a preset dual-stream network model; wherein the preset dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same;

[0007] Performing a first preliminary feature extraction on the original visible light image through the first network model to obtain a first preliminary feature image, and performing a first key feature extraction on the first preliminary feature image to obtain a first feature image; wherein the first feature image is used to characterize the features of the preset target;

[0008] Performing a second preliminary feature extraction on the original infrared image through the second network model to obtain a second preliminary feature image, and performing a second key feature extraction on the second preliminary feature image to obtain a second feature image; wherein the second feature image is used to characterize the features of the preset target;

[0009] Performing cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image; wherein the first fused image is used to characterize the features of the preset target, and the second fused image is used to characterize the features of the preset target;

[0010] The dual-stream network model is trained according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model; wherein the target recognition model is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by photographing the first target and a target infrared image obtained by photographing the second target, and the target recognition result includes:

[0011] a first result in which the first objective and the second objective are the same;

[0012] or

[0013] A second result in which the first objective is different from the second objective.

[0014] In some embodiments, extracting a first key feature from the first preliminary feature image to obtain a first feature image includes:

[0015] Performing self-attention enhancement processing on the first preliminary feature image to obtain a first enhanced image;

[0016] Perform convolution processing on the first enhanced image to obtain the first feature image.

[0017] In some embodiments, performing self-attention enhancement processing on the first preliminary feature image to obtain a first enhanced image includes:

[0018] Performing aggregation processing on the first preliminary feature image to obtain an aggregated image;

[0019] Performing normalized activation processing on the aggregated image to obtain a channel activation image;

[0020] Adjusting the channel activation image to obtain a channel attention image;

[0021] compressing the first preliminary feature image to obtain a compressed image;

[0022] Performing normalized activation processing on the compressed image to obtain a spatially activated image;

[0023] Adjusting the spatial activation image to obtain a spatial attention image;

[0024] The first enhanced image is obtained according to the first preliminary feature image, the channel attention image and the spatial attention image.

[0025] In some embodiments, the adjusting the channel activation image to obtain the channel attention image includes:

[0026] Get the preset response threshold;

[0027] Obtaining pixel values ​​of the channel activation image to obtain channel image pixel values;

[0028] According to the response threshold and the channel image pixel value, the channel activation image is enhanced or weakened to obtain the channel attention image.

[0029] In some embodiments, the step of enhancing or weakening the channel activation image according to the response threshold and the channel image pixel value to obtain the channel attention image includes:

[0030] Get the preset strengthening factor, get the preset weakening factor;

[0031] If the response threshold is greater than or equal to the channel image pixel value, the region corresponding to the channel image pixel value is enhanced by the enhancement factor to obtain the channel attention image;

[0032] If the response threshold is less than the channel image pixel value, the area corresponding to the channel image pixel value is weakened by the weakening factor to obtain the channel attention image.

[0033] In some embodiments, training the two-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain the target recognition model includes:

[0034] Calculating a first loss value according to the first feature image, the second feature image, the first fused image, and the second fused image;

[0035] Calculating a second loss value according to the first feature image and the second feature image;

[0036] Calculating a target loss value according to the first loss value and the second loss value;

[0037] The parameters of the dual-stream network model are updated according to the target loss value to obtain the target recognition model.

[0038] To achieve the above object, a second aspect of an embodiment of the present application provides a target recognition method, characterized in that the target recognition method includes:

[0039] Acquire a target image; wherein the target image includes a target visible light image and a target infrared image, the target visible light image is obtained by photographing a first target, and the target infrared image is obtained by photographing a second target;

[0040] Inputting the target visible light image and the target infrared image into a target recognition model; wherein the target recognition model is trained using the model training method described in any one of the embodiments of the first aspect;

[0041] The target visible light image and the target infrared image are recognized by the target recognition model to obtain a target recognition result; wherein the target recognition result includes:

[0042] a first result in which the first objective and the second objective are the same;

[0043] or

[0044] A second result in which the first objective is different from the second objective.

[0045] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a model training device, an acquisition module, for acquiring an original training image; wherein the original training image includes an original infrared image and an original visible light image, the original infrared image is obtained by photographing a preset target, and the original visible light image is obtained by photographing the preset target;

[0046] An image input module, used to input the original training image into a preset two-stream network model; wherein the preset two-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same;

[0047] A visible light extraction module, configured to perform a first preliminary feature extraction on the original visible light image through the first network model to obtain a first preliminary feature image, and perform a first key feature extraction on the first preliminary feature image to obtain a first feature image; wherein the first feature image is used to characterize the features of the preset target;

[0048] an infrared extraction module, configured to perform a second preliminary feature extraction on the original infrared image through the second network model to obtain a second preliminary feature image, and perform a second key feature extraction on the second preliminary feature image to obtain a second feature image; wherein the second feature image is used to characterize the features of the preset target;

[0049] A fusion processing module, used to perform cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image; wherein the first fused image is used to characterize the characteristics of the preset target, and the second fused image is used to characterize the characteristics of the preset target;

[0050] A training processing module is used to train the two-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model; wherein the target recognition model is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by photographing the first target and a target infrared image obtained by photographing the second target, and the target recognition result includes:

[0051] a first result in which the first objective and the second objective are the same;

[0052] or

[0053] A second result in which the first objective is different from the second objective.

[0054] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the program is executed by the processor, the method described in the first aspect is implemented; or,

[0055] Implement the steps of the target identification method as described in the embodiment of the second aspect.

[0056] To achieve the above-mentioned purpose, the fifth aspect of the embodiments of the present application proposes a storage medium, wherein the storage medium is a computer-readable storage medium, used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above; or,

[0057] Implement the steps of the target identification method as described in the embodiment of the second aspect.

[0058] The present application proposes a model training method and device, target recognition method, equipment, and medium, which obtain an original training image, wherein the original training image includes an original infrared image and an original visible light image; then the original training image is input into a preset dual-stream network model, wherein the dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same; then the first network model is used to perform a first preliminary feature extraction on the original visible light image to obtain a first preliminary feature image, and a first key feature extraction is performed on the first preliminary feature image to obtain a first feature image. In this way, the specific features of the original visible light image can be first extracted, and then the key features common to the original visible light image and the original infrared image can be extracted; then the second network model is used to perform a second preliminary feature extraction on the original infrared image to obtain a second preliminary feature image, and a second key feature extraction is performed on the second preliminary image. Take, get the second feature image, such setting, can first extract the specific features of the original infrared image, and then extract the key features common to the original visible light image and the original infrared image; then perform cross-modal fusion processing on the first feature image and the second feature image to obtain the first fused image and the second fused image, wherein the first fused image is used to characterize the features of the preset target, and the second fused image is used to characterize the features of the preset target, and finally, train the two-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model, which is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by shooting the first target and a target infrared image obtained by shooting the second target, and the target recognition result includes: a first result that the first target and the second target are the same; or a second result that the first target and the second target are different. By such setting, the accuracy of the model in recognizing the target image can be improved.

[0059] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0061] Figure 1 is a flow chart of a training method for a model provided in an embodiment of the present application;

[0062] Figure 2 yes Figure 1A flowchart of the specific method of step S300 in FIG.

[0063] Figure 3 yes Figure 2 A flowchart of the specific method of step S310 in FIG.

[0064] Figure 4 yes Figure 3 A flowchart of the specific method of step S313 in FIG.

[0065] Figure 5 yes Figure 4 A flowchart of the specific method of step S3133 in FIG.

[0066] Figure 6 yes Figure 1 A flowchart of the specific method of step S600 in FIG.

[0067] Figure 7 yes Figure 6 A flowchart of the specific method of step S610 in FIG.

[0068] Figure 8 It is a schematic diagram of a specific application scenario of the training method of the model provided in the embodiment of the present application;

[0069] Fig. 9 is a schematic diagram of the structure of the adaptive attention module provided in an embodiment of the present application;

[0070] Fig.10 is a flow chart of a target recognition method provided by an embodiment of the present application;

[0071] Fig.11 It is a structural schematic diagram of a training device for a model provided in an embodiment of the present application;

[0072] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0076] First, some nouns involved in this application are analyzed:

[0077] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0078] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0079] Relative entropy: Relative entropy is also called Kullback-Leibler divergence (KL divergence) or information divergence. It is an asymmetric measure of the difference between two probability distributions. In information theory, relative entropy is equivalent to the difference between the information entropies (Shannon entropy) of two probability distributions.

[0080] Person Re-Identification (Re-ID) aims to retrieve images of a given pedestrian from non-overlapping cameras. This technology plays an important role in intelligent surveillance and personnel tracking.

[0081] Object recognition is usually used to identify related objects (such as pedestrians, vehicles, faces, etc.), such as object re-identification, where the general principle of object re-identification is: given a preset image including a target, find an image of the same target as the preset image from the image data set; in related technologies, the recognition accuracy of object re-identification is not high. Therefore, how to improve the accuracy of object recognition has become a technical problem that people skilled in the art need to solve urgently.

[0082] Based on this, the embodiments of the present application provide a model training method and device, a target recognition method, equipment, and medium, aiming to improve the accuracy of target recognition.

[0083] The embodiments of the present application provide a model training method and device, target recognition method, equipment, and medium, which are specifically illustrated through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0084] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0085] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0086] The training method of the model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The training method of the model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. Cloud servers; software can be applications that implement model training methods, etc., but are not limited to the above forms.

[0087] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0088] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0089] Figure 1 is an optional flowchart of the training method of the model provided in the embodiment of the present application, Figure 1 The method may include steps S100 to S600. It should be understood that the training method of the model in the embodiment of the present application includes but is not limited to steps S100 to S600. Figure 1 These six steps are introduced in detail.

[0090] Step S100, obtaining an original training image; wherein the original training image includes an original infrared image and an original visible light image, the original infrared image is obtained by photographing a preset target, and the original visible light image is obtained by photographing a preset target;

[0091] Step S200, inputting the original training image into a preset dual-stream network model; wherein the preset dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same;

[0092] Step S300, performing a first preliminary feature extraction on the original visible light image through a first network model to obtain a first preliminary feature image, and performing a first key feature extraction on the first preliminary feature image to obtain a first feature image; wherein the first feature image is used to characterize the features of a preset target;

[0093] Step S400, performing a second preliminary feature extraction on the original infrared image through a second network model to obtain a second preliminary feature image, and performing a second key feature extraction on the second preliminary feature image to obtain a second feature image; wherein the second feature image is used to characterize the features of a preset target;

[0094] Step S500, performing cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image; wherein the first fused image is used to characterize the characteristics of a preset target, and the second fused image is used to characterize the characteristics of the preset target;

[0095] Step S600, training the dual-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model; wherein the target recognition model is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by photographing the first target and a target infrared image obtained by photographing the second target, and the target recognition result includes: a first result that the first target is identical to the second target; or a second result that the first target is different from the second target.

[0096] The training method of the model of the embodiment of the present application obtains an original training image, wherein the original training image includes an original infrared image and an original visible light image; then the original training image is input into a preset dual-stream network model, wherein the dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same; then the first network model is used to perform a first preliminary feature extraction on the original visible light image to obtain a first preliminary feature image, and the first key feature extraction is performed on the first preliminary feature image to obtain a first feature image. In this way, the specific features of the original visible light image can be first extracted, and then the key features common to the original visible light image and the original infrared image can be extracted; then the second network model is used to perform a second preliminary feature extraction on the original infrared image to obtain a second preliminary feature image, and the second key feature extraction is performed on the second preliminary image to obtain a second feature image. For example, by setting it this way, it is possible to first extract the specific features of the original infrared image, and then extract the key features common to the original visible light image and the original infrared image; then perform cross-modal fusion processing on the first feature image and the second feature image to obtain the first fused image and the second fused image, wherein the first fused image is used to characterize the features of the preset target, and the second fused image is used to characterize the features of the preset target; finally, the double-stream network model is trained according to the first feature image, the second feature image, the first fused image, and the second fused image to obtain a target recognition model, which is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by shooting the first target and a target infrared image obtained by shooting the second target, and the target recognition result includes: a first result in which the first target is the same as the second target; or a second result in which the first target is different from the second target. By setting it this way, the accuracy of the model in recognizing the target image can be improved.

[0097] In step S100 of some embodiments, the original training image includes an original infrared image and an original visible light image. The original infrared image can be obtained by shooting a preset target with an infrared camera or by other means, and the original visible light image can be obtained by shooting a preset target with an ordinary camera or by other means. The original training image of the embodiment of the present application can also be obtained through cloud service, and the present application does not make any specific restrictions on this.

[0098] It should be noted that the technical solution of the embodiment of the present application is not only applicable to the combination of the original visible light image and the original infrared image, but also to other types of image modality combinations, such as the combination of a black and white image and an original infrared image.

[0099] It should be noted that the preset target may be a pedestrian, or other objects, such as an animal, etc. If the preset target is a pedestrian, the technical solution of the embodiment of the present application may be applied in the field of pedestrian re-identification technology.

[0100] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0101] In step S200 of some embodiments, ResNet50 pre-trained on ImageNet can be used, and then the dual-stream network structure design in the shared feature method can be adopted, and then the full connection layer and average pooling layer after ResNet50 can be removed to obtain the dual-stream network model in the embodiment of the present application. It can be understood that the dual-stream network model in the embodiment of the present application can also be other designs, and this application does not make specific restrictions on this.

[0102] In step S300 of some embodiments, the first network model includes a first preliminary extraction module and a first key extraction module; wherein the first preliminary extraction module includes a convolution layer for performing a first preliminary feature extraction on the original visible light image to obtain a first preliminary feature image. For example, the feature extraction of the original visible light image can be performed through two layers of convolution layers to obtain a first preliminary feature image. With such a setting, specific features of the original visible light image can be extracted. After obtaining the first preliminary feature image, the first key extraction module performs a first key feature extraction on the first preliminary feature image to obtain the first feature image. By performing the first key feature extraction on the first preliminary feature image, the key features of the original visible light image can be extracted.

[0103] For example, in the field of pedestrian re-identification, the preset target is a pedestrian, and the original visible light image and the original infrared image are obtained by taking photos of the pedestrian.

[0104] It should be understood that the infrared image directly reflects the temperature distribution of the surface of the object, but because the infrared radiation of the target is very complex and there are many factors that affect the infrared radiation of the target, the clarity of the infrared thermal image is far inferior to that of the visible light image. The visible light image can well depict the shape structure of each object in the scene and has good contour expression. Therefore, the processing of the original visible light image in this scheme is: first, the first preliminary feature extraction is performed on the original visible light image through the first preliminary extraction module (with two convolution layers) to obtain the first preliminary feature image, and the first preliminary feature extraction is to extract the specific features of the original visible light image, such as: clear contour, etc. Then, the first key feature extraction is performed on the first preliminary feature image through the first key extraction module to obtain the first feature image, and the first key feature extraction is to extract the key features of the original visible light image, or extract the common features of the original visible light image and the infrared image, such as: the original visible light image and the original infrared image are obtained by shooting the same pedestrian, then the original visible light image and the original infrared image must have the same features.

[0105] See also Figure 2 In some embodiments, the step of "extracting the first key feature from the first preliminary feature image to obtain the first feature image" in step S300 includes steps S310 and S320. It should be understood that step S300 includes but is not limited to steps S310 and S320. Figure 2 These two steps are introduced in detail.

[0106] Step S310, performing self-attention enhancement processing on the first preliminary feature image to obtain a first enhanced image;

[0107] Step S320: performing convolution processing on the first enhanced image to obtain a first feature image.

[0108] For details, please refer to Figure 3 In some embodiments, step S310 includes step S311, step S312, step S313, step S314, step S315, step S316 and step S317. It should be understood that step S310 includes but is not limited to step S311 to step S317. Figure 3 These seven steps are introduced in detail.

[0109] Step S311, performing aggregation processing on the first preliminary feature image to obtain an aggregated image;

[0110] Step S312, performing normalization activation processing on the aggregated image to obtain a channel activation image;

[0111] Step S313, adjusting the channel activation image to obtain a channel attention image;

[0112] Step S314, compressing the first preliminary feature image to obtain a compressed image;

[0113] Step S315, performing normalization activation processing on the compressed image to obtain a spatial activation image;

[0114] Step S316, adjusting the spatial activation image to obtain a spatial attention image;

[0115] Step S317, obtaining a first enhanced image according to the first preliminary feature image, the channel attention image and the spatial attention image.

[0116] Specifically, in this implementation, the self-attention enhancement process includes channel attention enhancement process and spatial attention enhancement process. Assume that the first preliminary feature image is represented by F, F∈R C×H×W Among them, H, W, and C are the height, width, and number of channels of the first preliminary feature image, respectively.

[0117] The process of channel attention enhancement is as follows: first, the first preliminary feature image is sent to the global maximum pooling layer (Global Max Pooling, GMP) and the average pooling layer (Global Average Pooling, GAP) respectively to realize the aggregation processing of the first preliminary feature image, and the result of the first preliminary feature image processed by GMP and the result of the first preliminary feature image processed by GAP are added to obtain an aggregated image; then the obtained aggregated image is normalized and activated using the sigmoid activation function to obtain a channel activation image, which characterizes the channel response distribution of the first preliminary feature image. Then, the obtained channel activation image is adjusted to obtain a channel attention image, thereby realizing the channel attention enhancement processing of the first preliminary feature image.

[0118] The process of spatial attention enhancement is as follows: first, compress the first preliminary feature image into a one-dimensional real number in the spatial dimension direction, then multiply and weight it to each channel, and then compress it along the channel dimension direction, thereby realizing the compression processing of the first preliminary feature image and obtaining a compressed image; then, the obtained compressed image is normalized and activated using the sigmoid activation function to obtain a spatial activation image, which characterizes the spatial response distribution of the first preliminary feature image. Then, the spatial activation image is adjusted to obtain a spatial attention image, thereby realizing the spatial attention enhancement processing of the first preliminary feature image.

[0119] After obtaining the channel attention image and the spatial attention image through the above operations, the first preliminary feature image, the channel attention image and the spatial attention image are multiplied, and then an instance normalization (IN) layer is passed to perform style normalization on the first preliminary feature image F to obtain the first enhanced image.

[0120] Please refer to Figure 4 In some embodiments, step S313 includes step S3131, step S3132 and step S3133. It should be understood that step S313 includes but is not limited to step S3131 to step S3133. Figure 4 These three steps are introduced in detail.

[0121] Step S3131, obtaining a preset response threshold;

[0122] Step S3132, obtaining the pixel value of the channel activation image to obtain the channel image pixel value;

[0123] Step S3133, based on the response threshold and the channel image pixel value, the channel activation image is enhanced or weakened to obtain a channel attention image.

[0124] Specifically, in this embodiment, a response threshold is set, and then the channel activation image is adjusted according to the response threshold and the channel image pixel value, so as to obtain a channel attention image.

[0125] Please refer to Figure 5 In some embodiments, step S3133 includes but is not limited to step S31331 to step S31333. Figure 5 These three steps are introduced in detail.

[0126] Step S31331, obtaining a preset strengthening factor and a preset weakening factor;

[0127] Step S31332: if the response threshold is greater than or equal to the channel image pixel value, the region corresponding to the channel image pixel value is enhanced by the enhancement factor to obtain a channel attention image;

[0128] Step S31333: if the response threshold is less than the channel image pixel value, the area corresponding to the channel image pixel value is weakened by the weakening factor to obtain a channel attention image.

[0129] Specifically, in this embodiment, the channel activation image is adjusted by setting the response threshold, the enhancement factor and the weakening factor to obtain the channel attention image. If the pixel value of a certain area of ​​the channel activation image is greater than or equal to the response threshold, the area is identified as a high response area, and the enhancement factor should be multiplied to increase the attention to the area; if the pixel value of a certain area of ​​the channel activation image is less than the response threshold, the area is identified as a low response area, and the weakening factor should be multiplied to weaken the attention to the area.

[0130] It should be noted that when the spatial activation image is adjusted to obtain the spatial attention image, the entire adjustment method is similar to the method of adjusting the channel activation image to obtain the channel attention image. For the specific operation process, please refer to the aforementioned channel activation image adjustment process, which will not be repeated here.

[0131] In step S320 of some embodiments, the first enhanced image obtained in the above step is convolved through a convolution layer to obtain a first feature image.

[0132] It should be noted that, in the embodiment of the present application, multiple first key extraction modules can be set, that is, the first key feature extraction of the original visible light image can be performed multiple times. For example, after obtaining the first feature image, the first feature image is subjected to self-attention enhancement processing, and then convolved through a convolution layer to obtain the first feature image after two first key feature extractions.

[0133] In step S400 of some embodiments, the processing method of the original infrared image is similar to step S330, and the second network model includes a second preliminary extraction module and a second key extraction module; wherein the second preliminary extraction module includes a convolution layer, which is used to perform a second preliminary feature extraction on the original infrared image to obtain a second preliminary feature image. For example, the feature extraction of the original infrared image can be performed through two layers of convolution layers to obtain a second preliminary feature image. With such a setting, the specific features of the original infrared image can be extracted. After obtaining the second preliminary feature image, the second key extraction module performs a second key feature extraction on the second preliminary feature image to obtain a second feature image. By performing the second key feature extraction on the second preliminary feature image, the key features of the original infrared image can be extracted.

[0134] It should be noted that the specific operation steps of step S400 in the embodiment of the present application are similar to step S300 and will not be repeated here.

[0135] It should be noted that the first network model and the second network model form a dual-stream network model, and the structures of the first network model and the second network model are exactly the same, but the parameters of the first preliminary extraction module in the first network model and the second preliminary extraction module in the second network model are not shared, and the parameters of the first preliminary key module in the first network model and the second key extraction module in the second network model are shared. That is, when the first preliminary feature is extracted and the second preliminary feature is extracted, the parameters of the two are not shared; when the first key feature is extracted and the second key feature is extracted, the parameters of the two are shared.

[0136] In step S500 of some embodiments, the first feature image is firstly cut into upper and lower parts to obtain a first feature upper image and a first feature lower image, and the second feature image is cut into the same way and proportion to obtain a second feature upper image and a second feature lower image, and then the first feature upper image and the second feature lower image are spliced ​​to obtain a first fused image, and the first feature lower image and the second feature upper image are spliced ​​to obtain a second fused image. By such an arrangement, the first fused image and the second fused image both have the features of the first feature image and the second feature image.

[0137] It should be noted that, in cross-modal processing, the segmentation method is not limited to upper and lower segmentation, but can also be left and right, etc. For this, the embodiments of the present application do not make specific limitations.

[0138] Please refer to Figure 6 In some embodiments, step S600 includes but is not limited to steps S610 to S640. Figure 6 These four steps are introduced in detail.

[0139] Step S610, calculating a first loss value according to the first feature image, the second feature image, the first fused image, and the second fused image;

[0140] Step S620, calculating a second loss value according to the first feature image and the second feature image;

[0141] Step S630, calculating a target loss value according to the first loss value and the second loss value;

[0142] Step S640, updating the parameters of the dual-stream network model according to the target loss value to obtain a target recognition model.

[0143] Please refer to Figure 7 In some embodiments, step S610 includes but is not limited to steps S611 to S614. Figure 7 These four steps are introduced in detail.

[0144] Step S611, performing global average pooling processing on the first feature image to obtain a first pooling vector, performing global average pooling processing on the second feature image to obtain a second pooling vector, performing global average pooling processing on the first fused image to obtain a third pooling vector, and performing global average pooling processing on the second fused image to obtain a fourth pooling vector;

[0145] Step S612, performing full connection processing on the first pooled vector to obtain a first prediction value, performing full connection processing on the second pooled vector to obtain a second prediction value, performing full connection processing on the third pooled vector to obtain a third prediction value, and performing full connection processing on the fourth pooled vector to obtain a fourth prediction value;

[0146] Step S613, performing normalization conversion processing on the first prediction value to obtain a first probability value, performing normalization conversion processing on the second prediction value to obtain a second probability value, performing normalization conversion processing on the third prediction value to obtain a third probability value, and performing normalization conversion processing on the fourth prediction value to obtain a fourth probability value;

[0147] Step S614: Calculate a first loss value according to the first probability value, the second probability value, the third probability value and the fourth probability value.

[0148] Specifically, in this embodiment, the first feature image, the second feature image, the first fused image and the second fused image are input to the global average pooling layer, and the global average pooling process is performed to obtain the first pooling vector, the second pooling vector, the third pooling vector and the fourth pooling vector respectively. Then, the four pooling vectors are respectively sent to the fully connected layer for classification prediction to obtain the first prediction value, the second prediction value, the third prediction value and the fourth prediction value; then, the softmax function is used for normalization conversion processing to obtain the first probability value P1, the second probability value P2, the third probability value P3 and the fourth probability value P4. Finally, the first loss value is calculated by formula (1), and formula (1) is specifically:

[0149]

[0150] In formula (1), D KL represents KL divergence, L KL Represents the first calculated loss value.

[0151] In step S620 of some embodiments, the first feature image and the second feature image are respectively subjected to global average pooling processing by GAP to obtain a first pooled feature image and a second pooled feature image. After obtaining the first pooled feature image and the second pooled feature image, on the one hand, the first pooled feature image and the second pooled feature image are subjected to Bi-directional Dual-constrained Top-Ranking (BDTR) loss metric learning, and the metric loss value is calculated; on the other hand, the first pooled feature image and the second pooled feature image are respectively input into the fully connected layer, and then the first classification loss value of the first pooled feature image and the second classification loss value of the second pooled feature image are obtained by the softmax function, and then the first classification loss value and the second classification loss value are added to obtain the target classification loss value; finally, the target classification loss value and the metric loss value are added to obtain the second loss value.

[0152] For the calculation of the loss value of the softmax function, formula (2) can be used. Formula (2) is specifically:

[0153]

[0154] In formula (2), N is the total number of categories of preset targets in the original training image, and p i is the predicted probability that the preset target in the original training image belongs to the i-th category, and y is the true label of the preset target.

[0155] Bi-directional Dual-constrained Top-Ranking (BDTR) loss metric learning is performed on the first pooled feature image and the second pooled feature image. Assume that the input anchor sample (first pooled feature image) and The vectors extracted from (the second pooled feature image) are and Its corresponding hard positive sample and Hard negative examples and The BDTR loss function is defined as follows:

[0156]

[0157] In formula (3), m is the threshold hyperparameter. D(*) is the distance function, where squared Euclidean distance is used. express.

[0158] In step S630 and step S640 of some embodiments, the target loss value is the sum of the first loss value and the second loss value. After obtaining the target loss value, the parameters of the dual-stream network model are updated according to the target loss value, thereby obtaining the target recognition model.

[0159] Please refer to Figure 8 and Fig. 9 The technical solution of the present application is described in detail below with a specific embodiment.

[0160] In this implementation, the backbone network uses ResNet50 pre-trained on ImageNet, and adopts the basic two-stream network structure design in the shared feature method. That is, in the embodiment of the present application, the two-stream network model adopts ResNet50, which includes a convolutional layer and an adaptive attention module (Adaptiveattention module, AAM). The architecture of the first network model and the second network model is exactly the same, except that the weight parameters of the convolutional layers of the first two layers are not shared, and the weight parameters of the subsequent layers are shared. In this embodiment, the first network model is provided with 5 convolutional layers, and the adaptive attention is respectively embedded in front of the third, fourth, and fifth layers of the backbone network.

[0161] Taking the field of pedestrian re-identification as an example, the original visible light image and the original infrared image are obtained by shooting. The original visible light image is input into the first network model for the first preliminary feature extraction and the first key feature extraction to obtain the first feature image; the original infrared image is input into the second network model for the second preliminary feature extraction and the second key feature extraction to obtain the second feature image. Among them, the first preliminary feature extraction and the second preliminary feature extraction are both realized through two layers of convolutional layers; the first key feature extraction is first enhanced by an adaptive attention model, and then passed through a convolutional layer, and then the convolved image is passed through an adaptive attention module and a convolutional layer, and finally, the adaptive attention model is strengthened and convolved by the convolutional layer to obtain the first feature image; the second key feature extraction is similar to the first key feature extraction to obtain the second feature image.

[0162] After obtaining the first feature image and the second feature image, on the one hand, the first feature image and the second feature image are cross-modally fused and KL divergence is calculated to obtain the first loss value. On the other hand, the first feature image and the second feature image are both passed through the global average pooling layer to obtain the first pooled feature image and the second pooled feature image. After obtaining the first pooled feature image and the second pooled feature image, on the one hand, the first pooled feature image and the second pooled feature image are subjected to Bi-directional Dual-constrained Top-Ranking (BDTR) loss and the metric loss value is calculated. On the other hand, the first pooled feature image and the second pooled feature image are respectively passed through the fully connected layer and the softmax function to calculate the target classification loss value, thereby obtaining the second loss value according to the target classification loss value and the metric loss value. Finally, the first loss value and the second loss value are added to obtain the target loss value to update the parameters of the dual-stream network model and obtain the target recognition model.

[0163] Please refer to Fig. 9 ,The adaptive attention module includes two parts: spatial attention and channel attention mechanism. The original visible light image is used as an example to illustrate. Assume that the first preliminary feature image of the input is represented by F, F∈R C×H×W Among them, H, W, and C are the height, width, and number of channels of the first preliminary feature image, respectively.

[0164] In the spatial attention branch, the first preliminary feature image is first compressed into a one-dimensional real number in the spatial dimension direction, then weighted to each channel by multiplication, and then compressed along the channel dimension direction, thereby realizing the compression processing of the first preliminary feature image and obtaining a compressed image; then, the obtained compressed image is normalized and activated using the sigmoid activation function to obtain a spatial activation image, which characterizes the spatial response distribution of the first preliminary feature image. The spatial activation image is then adjusted to obtain a spatial attention image, thereby realizing the spatial attention enhancement processing of the first preliminary feature image. This can be briefly described by formulas (4) to (6), which are as follows:

[0165]

[0166]

[0167] U=Sigmoid(W) (6)

[0168] In formula (4), i is a natural number between 1 and H, and j is a natural number between 1 and W. In formula (5), k is a natural number between 1 and C.

[0169] In the channel attention branch, the first preliminary feature image is first sent to the global maximum pooling layer and the average pooling layer respectively to realize the aggregation processing of the first preliminary feature image, and the result of the first preliminary feature image processed by GMP and the result of the first preliminary feature image processed by GAP are added to obtain an aggregated image; the obtained aggregated image is then normalized and activated using the sigmoid activation function to obtain a channel activation image, which characterizes the channel response distribution of the first preliminary feature image. The obtained channel activation image is then adjusted to obtain a channel attention image, thereby realizing the channel attention enhancement processing of the first preliminary feature image. It can be briefly described by formula (7) and formula (8), and formula (7) and (8) are as follows:

[0170] f=GAP(F)+GMP(F) (7)

[0171] Q=Sigmoid(f) (8)

[0172] Through the above operations, the spatial response distribution and channel response distribution of the first preliminary feature image are obtained. Among them, the area with high pixel values ​​represents that the two-stream network model pays more attention to the high response area of ​​the first preliminary image, while the area with low values ​​indicates that the corresponding area is rarely paid attention to or not paid attention to. By setting the response threshold, strengthening factor and weakening factor to regulate the spatial and channel attention, the pixel values ​​in U and Q are greater than the response threshold β h The area is considered to be a high response area and should be multiplied by the enhancement factor α h , increasing the emphasis on these areas; and the value is less than the response threshold β l The area where is considered to be a low response area is multiplied by the attenuation factor α l , weakening the attention to this area. The impact factors of other areas are 1, that is, remain unchanged. The specific operation is shown in formula (9) and formula (10), which are:

[0173]

[0174]

[0175] After obtaining the channel attention image and the spatial attention image through the above operations, the first preliminary feature image, the channel attention image and the spatial attention image are multiplied together, and then the first preliminary feature image F is style normalized through an instance normalization layer to obtain the first enhanced image.

[0176] It should be noted that since the adaptive attention module and cross-modal feature fusion mechanism are designed to improve the ability of the two-stream network model to capture distinguishing information and thus identify targets of different modalities, they are used in the training phase to optimize the two-stream network model, and the two-stream network model has the ability to extract features, so the adaptive attention module and cross-modal feature fusion mechanism only work in the training phase. In the testing phase, it is only necessary to enter the corresponding two-stream network model branch according to the modality of the input image. At this time, the two-stream network model branch has the ability to focus on key features after training, and the acquired feature vector is used as the final feature representation of the image corresponding to the preset target for subsequent cross-modal retrieval.

[0177] It should be noted that each source of information can be a modality, including but not limited to the following: human senses, including touch, hearing, vision, smell, and taste; information media, including voice, video, text, etc.; various sensors, including radar sensors, infrared sensors, acceleration sensors, etc. In the embodiment of the present application, different modalities refer to different image types of the input image. The modality of the input image can be an infrared image, a visible light image, a black and white image, etc. In this embodiment, the modality of the input image is any two of the above image types.

[0178] Please refer to Fig.10 In the second aspect, the present application embodiment also proposes a target recognition method, including but not limited to step S700 and step S800. Fig.10 These two steps are introduced in detail.

[0179] Step S700, acquiring a target image; wherein the target image includes a target visible light image and a target infrared image, the target visible light image is obtained by photographing a first target, and the target infrared image is obtained by photographing a second target;

[0180] Step S800: inputting the target visible light image and the target infrared image into a target recognition model; wherein the target recognition model is trained according to the model training method of any one of the embodiments of the first aspect;

[0181] Step S900, identifying the target visible light image and the target infrared image through the target recognition model to obtain a target recognition result; wherein the target recognition result includes: a first result that the first target and the second target are the same; or a second result that the first target and the second target are different.

[0182] The target recognition method of the embodiment of the present application uses the target recognition model obtained by the aforementioned method to perform recognition processing on the target image to obtain a target recognition result, thereby improving the accuracy of target image recognition.

[0183] It should be noted that in the embodiment of the present application, the first target and the second target are similar to the aforementioned preset targets, and can be pedestrians, animals, etc., and the present application does not impose any specific restrictions on this.

[0184] Please refer to Fig.11 In the third aspect, some embodiments of the present application also propose a model training device, including an acquisition module 1000, an image input module 1100, a visible light extraction module 1200, an infrared extraction module 1300, a fusion processing module 1400 and a training processing module 1500.

[0185] The acquisition module 1000 is used to acquire original training images; wherein the original training images include original infrared images and original visible light images, the original infrared images are obtained by photographing a preset target, and the original visible light images are obtained by photographing a preset target.

[0186] The image input module 1100 is used to input the original training image into a preset dual-stream network model; wherein the preset dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same.

[0187] The visible light extraction module 1200 is used to perform a first preliminary feature extraction on the original visible light image through a first network model to obtain a first preliminary feature image, and perform a first key feature extraction on the first preliminary feature image to obtain a first feature image; wherein the first feature image is used to characterize the characteristics of a preset target.

[0188] The infrared extraction module 1300 is used to perform a second preliminary feature extraction on the original infrared image through a second network model to obtain a second preliminary feature image, and perform a second key feature extraction on the second preliminary feature image to obtain a second feature image; wherein the second feature image is used to characterize the characteristics of a preset target.

[0189] The fusion processing module 1400 is used to perform cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image; wherein the first fused image is used to characterize the characteristics of a preset target, and the second fused image is used to characterize the characteristics of the preset target.

[0190] The training processing module 1500 is used to train the dual-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model; wherein the target recognition model is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by photographing the first target and a target infrared image obtained by photographing the second target, and the target recognition result includes: a first result that the first target is identical to the second target; or a second result that the first target is different from the second target.

[0191] The training device of the model of the embodiment of the present application obtains an original training image, wherein the original training image includes an original infrared image and an original visible light image; then the original training image is input into a preset dual-stream network model, wherein the dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same; then the first network model is used to perform a first preliminary feature extraction on the original visible light image to obtain a first preliminary feature image, and the first key feature extraction is performed on the first preliminary feature image to obtain a first feature image. In this way, the specific features of the original visible light image can be first extracted, and then the key features common to the original visible light image and the original infrared image can be extracted; then the second network model is used to perform a second preliminary feature extraction on the original infrared image to obtain a second preliminary feature image, and the second key feature extraction is performed on the second preliminary image to obtain a second feature image. For example, by setting it this way, it is possible to first extract the specific features of the original infrared image, and then extract the key features common to the original visible light image and the original infrared image; then perform cross-modal fusion processing on the first feature image and the second feature image to obtain the first fused image and the second fused image, wherein the first fused image is used to characterize the features of the preset target, and the second fused image is used to characterize the features of the preset target; finally, the double-stream network model is trained according to the first feature image, the second feature image, the first fused image, and the second fused image to obtain a target recognition model, which is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by shooting the first target and a target infrared image obtained by shooting the second target, and the target recognition result includes: a first result in which the first target is the same as the second target; or a second result in which the first target is different from the second target. By setting it this way, the accuracy of the model in recognizing the target image can be improved.

[0192] The specific implementation of the training device for this model is basically the same as the specific implementation of the training method for the above-mentioned model, and will not be repeated here.

[0193] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the training method of the above model or the target recognition method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0194] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0195] The processor 1600 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0196] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the training method of the model in the embodiment of this application;

[0197] Input / output interface 1800, used to implement information input and output;

[0198] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0199] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0200] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0201] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned model training method or the above-mentioned target recognition method.

[0202] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0204] It can be understood by those skilled in the art that Figure 1-7 and Fig.10 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0213] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A model training method, characterized in that: The training method of the model includes: Acquire an original training image; wherein the original training image includes an original infrared image and an original visible light image, the original infrared image is obtained by photographing a preset target, and the original visible light image is obtained by photographing the preset target; Inputting the original training image into a preset dual-stream network model; wherein the preset dual-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same; Performing a first preliminary feature extraction on the original visible light image through the first network model to obtain a first preliminary feature image, and performing a first key feature extraction on the first preliminary feature image to obtain a first feature image; wherein the first feature image is used to characterize the features of the preset target; Performing a second preliminary feature extraction on the original infrared image through the second network model to obtain a second preliminary feature image, and performing a second key feature extraction on the second preliminary feature image to obtain a second feature image; wherein the second feature image is used to characterize the features of the preset target; Performing cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image; wherein the first fused image is used to characterize the features of the preset target, and the second fused image is used to characterize the features of the preset target; The dual-stream network model is trained according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model; wherein the target recognition model is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by photographing the first target and a target infrared image obtained by photographing the second target, and the target recognition result includes: a first result in which the first objective and the second objective are the same; or a second result in which the first objective is different from the second objective; The step of training the two-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model includes: Calculating a first loss value according to the first feature image, the second feature image, the first fused image, and the second fused image; Calculating a second loss value according to the first feature image and the second feature image; Calculating a target loss value according to the first loss value and the second loss value; Updating the parameters of the dual-stream network model according to the target loss value to obtain the target recognition model; The first loss value is calculated by the following formula: , represents the first loss value, represents KL divergence, P1 is the first probability value, P2 is the second probability value, P3 is the third probability value, and P4 is the fourth probability value. The first probability value is obtained by performing global average pooling on the first feature image and then performing classification prediction through a fully connected layer. The second probability value is obtained by performing global average pooling on the second feature image and then performing classification prediction through a fully connected layer. The third probability value is obtained by performing global average pooling on the first fused image and then performing classification prediction through a fully connected layer. The fourth probability value is obtained by performing global average pooling on the second fused image and then performing classification prediction through a fully connected layer.

2. The method according to claim 1, characterized in that The step of extracting a first key feature from the first preliminary feature image to obtain a first feature image includes: Performing self-attention enhancement processing on the first preliminary feature image to obtain a first enhanced image; Perform convolution processing on the first enhanced image to obtain the first feature image.

3. The method according to claim 2, characterized in that The performing self-attention enhancement processing on the first preliminary feature image to obtain a first enhanced image includes: Performing aggregation processing on the first preliminary feature image to obtain an aggregated image; Performing normalized activation processing on the aggregated image to obtain a channel activation image; Adjusting the channel activation image to obtain a channel attention image; Compressing the first preliminary feature image to obtain a compressed image; Performing normalized activation processing on the compressed image to obtain a spatially activated image; Adjusting the spatial activation image to obtain a spatial attention image; The first enhanced image is obtained according to the first preliminary feature image, the channel attention image and the spatial attention image.

4. The method according to claim 3, characterized in that The adjusting process of the channel activation image to obtain the channel attention image includes: Get the preset response threshold; Obtaining pixel values ​​of the channel activation image to obtain channel image pixel values; According to the response threshold and the channel image pixel value, the channel activation image is enhanced or weakened to obtain the channel attention image.

5. The method according to claim 4, characterized in that The step of strengthening or weakening the channel activation image according to the response threshold and the channel image pixel value to obtain the channel attention image includes: Get the preset strengthening factor, get the preset weakening factor; If the response threshold is greater than or equal to the channel image pixel value, the region corresponding to the channel image pixel value is enhanced by the enhancement factor to obtain the channel attention image; If the response threshold is less than the channel image pixel value, the area corresponding to the channel image pixel value is weakened by the weakening factor to obtain the channel attention image.

6. A target recognition method, characterized in that: The target recognition method comprises: Acquire a target image; wherein the target image includes a target visible light image and a target infrared image, the target visible light image is obtained by photographing a first target, and the target infrared image is obtained by photographing a second target; Inputting the target visible light image and the target infrared image into a target recognition model; wherein the target recognition model is trained according to the model training method according to any one of claims 1 to 5; The target visible light image and the target infrared image are recognized by the target recognition model to obtain a target recognition result; wherein the target recognition result includes: a first result in which the first objective and the second objective are the same; or A second result in which the first objective is different from the second objective.

7. A model training device, characterized in that: The training device of the model includes: An acquisition module, used to acquire an original training image; wherein the original training image includes an original infrared image and an original visible light image, the original infrared image is obtained by photographing a preset target, and the original visible light image is obtained by photographing the preset target; An image input module, used to input the original training image into a preset two-stream network model; wherein the preset two-stream network model includes a first network model and a second network model, and the first network model and the second network model are the same; A visible light extraction module, configured to perform a first preliminary feature extraction on the original visible light image through the first network model to obtain a first preliminary feature image, and perform a first key feature extraction on the first preliminary feature image to obtain a first feature image; wherein the first feature image is used to characterize the features of the preset target; an infrared extraction module, configured to perform a second preliminary feature extraction on the original infrared image through the second network model to obtain a second preliminary feature image, and perform a second key feature extraction on the second preliminary feature image to obtain a second feature image; wherein the second feature image is used to characterize the features of the preset target; A fusion processing module, used to perform cross-modal fusion processing on the first feature image and the second feature image to obtain a first fused image and a second fused image; wherein the first fused image is used to characterize the characteristics of the preset target, and the second fused image is used to characterize the characteristics of the preset target; A training processing module is used to train the two-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model; wherein the target recognition model is used to perform recognition processing on the target image to obtain a target recognition result; the target image includes a target visible light image obtained by photographing the first target and a target infrared image obtained by photographing the second target, and the target recognition result includes: a first result in which the first objective and the second objective are the same; or a second result in which the first objective is different from the second objective; The training processing module is used to train the two-stream network model according to the first feature image, the second feature image, the first fused image and the second fused image to obtain a target recognition model, including: Calculating a first loss value according to the first feature image, the second feature image, the first fused image, and the second fused image; Calculating a second loss value according to the first feature image and the second feature image; Calculating a target loss value according to the first loss value and the second loss value; Updating the parameters of the dual-stream network model according to the target loss value to obtain the target recognition model; The first loss value is calculated by the following formula: , represents the first loss value, represents KL divergence, P1 is the first probability value, P2 is the second probability value, P3 is the third probability value, and P4 is the fourth probability value. The first probability value is obtained by performing global average pooling on the first feature image and then performing classification prediction through a fully connected layer. The second probability value is obtained by performing global average pooling on the second feature image and then performing classification prediction through a fully connected layer. The third probability value is obtained by performing global average pooling on the first fused image and then performing classification prediction through a fully connected layer. The fourth probability value is obtained by performing global average pooling on the second fused image and then performing classification prediction through a fully connected layer.

8. An electronic device, characterized in that: The electronic device comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the program is executed by the processor, the steps of the training method of the model as claimed in any one of claims 1 to 5 are realized; or, Implement the steps of the target recognition method as described in claim 6.

9. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the model training method described in any one of claims 1 to 5; or, Implement the steps of the target recognition method as described in claim 6.