Liveness detection method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for liveness detection, which includes: obtaining at least two modal images of an object to be detected; extracting first target information from the first modal image; the first modal image is one of the at least two modal images; based on the first target information, extracting second target information from the second modal image; the second modal image is an image of at least two modal images other than the first modal image; performing liveness detection based on fusion information of the first target information and the second target information to obtain a liveness detection result; the liveness detection result is used to characterize whether the object to be detected is alive. In the above manner, the present application can improve the accuracy of liveness detection.

Description

Technical Field

[0001] The present application relates to the field of target recognition technology, and in particular to a living body detection method, device and storage medium. Background Art

[0002] Target recognition technology can be applied to many fields such as gate access control, attendance machine, mobile phone face payment, etc. However, the current target recognition technology is vulnerable to the risk of disguised target attack, that is, the object of target recognition is not the image of the real target, but the image of the disguised target, such as the paper photo of the target, the electronic image of the target, the three-dimensional mold, etc.

[0003] Based on this, it is necessary to perform liveness detection on the target image during the target recognition process to determine whether the target is alive. It is of great significance to improve the accuracy of liveness detection results and avoid identifying disguised targets as real targets. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a liveness detection method, device and storage medium, which can improve the accuracy of liveness detection.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a liveness detection method, the method comprising: obtaining at least two modal images of an object to be detected; extracting first target information from the first modal image; the first modal image is one of the at least two modal images; based on the first target information, extracting second target information from the second modal image; the second modal image is an image of at least two modal images except the first modal image; performing liveness detection based on fusion information of the first target information and the second target information to obtain a liveness detection result; the liveness detection result is used to characterize whether the object to be detected is alive.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0007] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0008] In the above scheme, after acquiring at least two modal images of the object to be detected, one of the modal images is used as the first modal image, and the remaining modal image is used as the second modal image, and then the first target information is first extracted from the first modal image, and then based on the first target information, the second target information is extracted from the second modal image, and then based on the fusion information of the first target information and the second target information, liveness detection is performed to obtain a liveness detection result. The present application guides the extraction of the second target information of the second modal image through the first target information of the first modal image, so that the extracted first target information and the second target information are complementary, and then the fusion information of the complementary first target information and the second target information is used for liveness detection. Compared with the method of directly extracting the first target information and the second target information and using their fusion information for liveness detection, the above method of the present application can effectively reduce the influence of irrelevant information on the liveness detection result, thereby improving the accuracy of liveness detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flow chart of an embodiment of a liveness detection method provided by the present application;

[0010] Figure 2 yes Figure 1 The flowchart of step S12 is shown as an embodiment;

[0011] Figure 3 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0012] Figure 4 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0014] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0015] See also Figure 1 , Figure 1is a flow chart of an embodiment of the liveness detection method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0016] S11: Acquire at least two modality images of the object to be detected.

[0017] This embodiment is used to use the first target information of the first modality image to guide the extraction of the second target information of the other modality image, so that the extracted first target information and the second target information are complementary, thereby making the result obtained by the fusion information detection using the complementary first target information and the second target information more accurate.

[0018] In this embodiment, at least two modal images are acquired by using at least two acquisition units to acquire the object to be detected. The acquisition unit may be a visible light camera, a camera with near-infrared shooting function, an infrared thermal imager, or a binocular stereo vision system. To ensure the accuracy of subsequent detection results, the difference between the image acquisition moments corresponding to different modal images can be made less than a preset threshold, so that the objects to be detected acquired by the images acquired by different acquisition units are the same. Optionally, the acquisition moments of different modal images can be the same moment, or any moment when the difference is less than or equal to the preset threshold. The specific preset threshold can be determined according to actual needs and is not specifically limited here.

[0019] In one embodiment, at least two modal images include at least two of visible light images, near-infrared images, depth maps, and thermal imaging images, wherein when performing liveness detection, the number of various modal images is one frame, which can be a frame with the best quality among multiple frames of images collected, or can be a randomly collected frame.

[0020] S12: extracting first target information from a first modality image; the first modality image is one of at least two modality images.

[0021] In one embodiment, feature extraction may be performed directly on the first modality image to obtain the first target information.

[0022] In another embodiment, the first feature extraction may be performed on the first modality image to obtain the first modality feature, and then information extraction may be performed on the first modality feature to obtain the first target information.

[0023] In an optional implementation, a feature extraction module may be used to extract features from the first modality image to obtain first modality features, and then a gated recurrent network unit or a long short-term memory network may be used to extract information from the first modality features to obtain first target information. The feature extraction module may be, but is not limited to, a convolutional neural network, and may also be other networks capable of feature extraction.

[0024] In this embodiment, the first modality image may be any one of at least two modality images, or may be a specific modality image selected from at least two modality images. For example, the at least two modality images include a visible light image, a near infrared image, and a depth map. The first modality image may be any one of the three modality images, or the visible light image may be directly selected as the first modality image, and the first target information of the visible light image may be subsequently used to guide the information extraction of the near infrared image and the depth map. The specific first modality image may be determined according to actual needs, and is not specifically limited here.

[0025] S13: extracting second target information from a second modality image based on the first target information; the second modality image is an image of at least two modality images excluding the first modality image.

[0026] In one embodiment, the second modality image may be firstly subjected to feature extraction to obtain the second modality feature, and then the first target information and the second modality feature may be subjected to preset processing to determine the retained data of the first target information and the impact data of the first target information on the second target information; and then the second modality feature, the retained data and the impact data may be used to extract the second target information from the second modality feature. The specific implementation process may be referred to below. Figure 2 Description of the illustrated embodiment.

[0027] Of course, in some embodiments, it is also possible to only determine the impact data of the first target information on the second target information, and use the second modal features and the impact data of the first target information on the second target information to extract the second target information from the second modal features, so as to use the impact data of the first target information on the second target information to guide the extraction of the second target information.

[0028] In one embodiment, the step of extracting features from the second modality image to obtain the second modality features can be performed by a feature extraction module, and the first target information and the second modality features are preset to determine the retained data of the first target information and the impact data of the first target information on the second target information; then the second modality features, the retained data and the impact data are used to extract the second target information from the second modality features, which is performed by a gated recurrent network unit or a long short-term memory network. It should be noted that the feature extraction module for extracting the second modality features and the feature extraction module for extracting the first modality features can be the same feature extraction module with shared weights, or different feature extraction modules that have been pre-trained. Among them, the two feature extraction modules can be but are not limited to convolutional neural networks; in addition, the gated recurrent network unit or long short-term memory network that executes to obtain the first target information and the second target information can be the same network, or different networks that are trained separately.

[0029] S14: Performing liveness detection based on the fusion information of the first target information and the second target information to obtain a liveness detection result; the liveness detection result is used to indicate whether the object to be detected is alive.

[0030] In one embodiment, after extracting the first target information and the second target information, the first target information and the second target information are directly fused to obtain fused information, and then liveness detection is performed based on the fused information to obtain a liveness detection result. The liveness detection result may be the probability of the object to be detected being classified as a live object and the probability of being classified as a non-live object, and then the result that meets the probability requirement is determined as the final detection result.

[0031] In another embodiment, after the first target information and the second target information are extracted, the second target information may be used to guide the extraction of the third target information. The third target information is information extracted from the first target information under the guidance of the second target information. The specific guidance method can refer to the above and Figure 2 Related description of the illustrated embodiment. After extracting the third target information, the second target information and the third target information may be fused to obtain fused information, and then liveness detection is performed to obtain liveness detection results.

[0032] In a specific embodiment, at least two modal images include a visible light image, a near-infrared image and a depth map, wherein one of the visible light image, the near-infrared image and the depth map can be used as a first modal image, and the other two can be used as second modal images. After extracting the first target information of the first modal image, based on the first target information, the second target information is extracted from each second modal image respectively, and then the first target information and the second target information corresponding to each second modal image are fused. After obtaining the fused information, liveness detection is performed to obtain the corresponding liveness detection result.

[0033] In the above scheme, after acquiring at least two modal images of the object to be detected, one of the modal images is used as the first modal image, and the remaining modal image is used as the second modal image, and then the first target information is first extracted from the first modal image, and then based on the first target information, the second target information is extracted from the second modal image, and then based on the fusion information of the first target information and the second target information, liveness detection is performed to obtain a liveness detection result. The present application guides the extraction of the second target information of the second modal image through the first target information of the first modal image, so that the extracted first target information and the second target information are complementary, and then the fusion information of the complementary first target information and the second target information is used for liveness detection. Compared with the method of directly extracting the first target information and the second target information and using their fusion information for liveness detection, the above method of the present application can effectively reduce the influence of irrelevant information on the liveness detection result, thereby improving the accuracy of liveness detection.

[0034] See also Figure 2 , Figure 2 yes Figure 1 The flowchart of step S12 of an embodiment is shown. In this embodiment, extracting the first target information from the first modality image includes:

[0035] S21: extracting features from the second modality image to obtain second modality features.

[0036] For details, please refer to the relevant description of step S12 above, and no further details will be given here.

[0037] S22: Preset processing is performed on the first target information and the second modal features to determine the retained data of the first target information and the impact data of the first target information on the second target information.

[0038] In this embodiment, the first target information and the second modal feature may be spliced ​​to obtain spliced ​​data, and then the spliced ​​data may be subjected to a first preset operation to obtain retained data, and the spliced ​​data may be subjected to a second preset operation to obtain impact data. The retained data is used to determine the amount of data of the first target information that needs to be retained when the second target information is subsequently extracted from the second modal feature, which determines the degree of retention of the first target information in the second target information; the impact data is used to determine how to fuse the second modal feature with the first target information when the second target information is extracted from the second modal feature, which determines which information in the first target information and the second modal feature is used as the second target information.

[0039] S23: extracting second target information from the second modal features using the second modal features, the retained data, and the impact data.

[0040] In this embodiment, a third preset operation may be performed on the second modal feature, the first target information and the influencing data to determine candidate information from the first target information; and then the second target information may be determined using the retained data, the first target information and the candidate information.

[0041] It can be seen from the above processing procedure that the second target information is determined based on the retained data in the first target information and the impact data on the second target information, that is, the present application can use the first target information to influence the information extraction of the second modality image, so that when extracting information from the second modality image, it can be guided by the extracted first target information to extract key and important information in the second modality image.

[0042] In a specific embodiment, the first target information and the second modal feature may be first spliced ​​to obtain spliced ​​data, and then a first preset operation may be performed on the spliced ​​data to obtain retained data, and a second preset operation may be performed on the spliced ​​data to determine influencing data, and then the second modal feature, the retained data and the influencing data may be used to extract the second target information from the second modal feature.

[0043] Exemplarily, please refer to the following formula to perform a first preset operation on the spliced ​​data to obtain the retained data, including: determining a first product of a first preset parameter and the spliced ​​data; and performing a first activation operation on the first product to obtain the retained data.

[0044]

[0045] In the formula, Indicates that data is retained. Indicates the first target information, represents the second modal feature, Represents concatenated data, is the first preset parameter, is the activation function, which is used for activation operation. Between 0 and 1, The closer it is to 1, the more first target information needs to be retained when extracting the second target information.

[0046] In this embodiment, please refer to the following formula to perform a second preset operation on the spliced ​​data to determine the influencing data, including: determining a second product of the second preset parameter and the spliced ​​data; and performing a second activation operation on the second product to obtain the influencing data.

[0047]

[0048] In the formula, Indicates the impact data, Represents concatenated data, represents the second preset parameter, Is the activation function, used for activation operation. Between 0 and 1, this value determines the extent to which the first target information should be retained when extracting the second target information. When it is close to 0, it means that we need to rely more on the current input information (second modal features) and ignore the previous first target information. When it is close to 1, it means that more information needs to be obtained from the first target information.

[0049] Further, please refer to the following formula and perform a third preset operation on the second modal feature, the first target information and the influencing data to determine the candidate information from the first target information, including: determining the third product of the first target information and the influencing data; obtaining the splicing result of the third product and the second modal feature, and performing a second activation operation on the splicing result and the fourth product of the third preset parameter to obtain the candidate information.

[0050]

[0051] In the formula, Indicates candidate information. represents the third product, represents the concatenation result of the third product and the second modal feature, represents the third preset parameter, () represents the activation function.

[0052] Furthermore, please refer to the following formula and use the retained data, the first target information and the candidate information to determine the second target information, including: using the retention degree as the first weight parameter, and using the difference between the target constant and the retention degree as the second weight parameter; determining the fifth product of the first weight parameter and the first target information, and determining the sixth product of the second weight parameter and the candidate information; and using the sum of the fifth product and the sixth product as the second target information.

[0053]

[0054] In the formula, the target constant is 1, Represents the second target information. The weighted sum of the first target information and the candidate information can be controlled by retaining data to determine the degree of retaining the first target information and the degree of introducing the second modal features, wherein, z i (Retained data) determines the degree of retention of the first target information and the degree of introduction of candidate information when extracting the second target information.

[0055] It can be understood that the value of ri can control the degree of integration of the first target information and the second modal features, and indirectly affect the calculation of the candidate hidden state. When the value of ri approaches 0, the candidate information relies more on the second modal features, and when it approaches 1, it relies more on the first target information. Furthermore, the retained data and the influencing data act together on the calculation of the candidate information and the second target information.

[0056] In another specific embodiment, the first target information and the second modal feature may be spliced ​​together to obtain spliced ​​data, and then the spliced ​​data may be subjected to a fourth preset operation to obtain retained data, and the spliced ​​data may be subjected to a fifth preset operation to determine influencing data, and the spliced ​​data may be subjected to a sixth preset operation to determine candidate memory elements, and then the candidate memory elements, the second modal feature, the retained data and the influencing data may be used to extract the second target information from the second modal feature.

[0057] Please refer to the following formula to perform the fourth preset operation on the spliced ​​data to obtain the retained data, including: respectively determining the seventh product of the fourth preset parameter and the spliced ​​data and the eighth product of the fifth preset parameter and the spliced ​​data; respectively performing the third activation operation on the seventh product and the eighth product to obtain the corresponding first retained data and second retained data.

[0058]

[0059]

[0060] In the formula, Indicates the first reserved data, Indicates the first target information, represents the second modal feature, Represents concatenated data, is the fourth preset parameter, Is the activation function, used for activation operation.

[0061] In this embodiment, please refer to the following formula to perform the fifth preset operation on the spliced ​​data to determine the influencing data, including: determining the ninth product of the sixth preset parameter and the spliced ​​data; performing the second activation operation on the ninth product to obtain the influencing data.

[0062]

[0063] In the formula, Indicates the impact data, Represents concatenated data, represents the sixth preset parameter, Is the activation function, used for activation operation. Between 0 and 1, this value determines which information in the first target information and the second modal features is used as the second target information when extracting the second target information.

[0064] Please refer to the following formula to perform the sixth preset operation on the spliced ​​data to determine the candidate memory element, including: determining the tenth product of the seventh preset parameter and the spliced ​​data; performing the third activation operation on the tenth product to obtain the candidate memory element.

[0065]

[0066] In the formula, represents the candidate memory element, Represents concatenated data, Indicates the seventh preset parameter, is an activation function used for performing activation operations. The candidate memory element is new memory information calculated according to the first target information and the second modal feature.

[0067] Further, please refer to the following formula, and utilize candidate memory elements, second modal features, retained data, and influence data to extract second target information from the second modal features, including: using the first retained data as the third weight and the second retained data as the fourth weight, determining the eleventh product of the first retention degree and the candidate memory element, and determining the twelfth product of the second retention degree and the first memory element of the first target information, and using the sum of the eleventh product and the twelfth product as the second memory element for determining the second target information; and then performing a fourth activation operation on the second memory element to obtain an activation operation result, and then performing a seventh preset operation on the activation operation result and the influence data to obtain the second target information.

[0068]

[0069]

[0070] In the formula, represents the second memory element, which is used to represent the memory information that is effective for the second target information. Indicates the second target information, represents the candidate memory element, represents the first memory element, Indicates the first reserved data, , Indicates the impact data, tanh Is the activation function, used for activation operation.

[0071] From the above, it can be seen that the first retained data is used to control how much new data from the candidate memory element is adopted, the second retained data is used to control how much data of the first memory element is retained, and the influencing data is used to transfer the effective second memory information to the second target information.

[0072] In one embodiment, the steps of presetting the first target information and the second modal features, determining the retained data of the first target information and the impact data of the first target information on the second target information, and extracting the second target information from the second modal features using the second modal features, the retained data and the impact data are performed by a recurrent network unit. The preset parameters are learning parameters of the recurrent network unit. The learning parameters are parameters learned during the training of the recurrent network unit.

[0073] In a specific embodiment, at least two modality images include a visible light image and a near infrared image. One of the visible light image and the near infrared image is used as the first modality image, and the other is used as the second modality image. Feature extraction is first performed on the first modality image and the second modality image respectively to obtain the first modality feature and the second modality feature. Then, the first modality feature is extracted using the first recurrent network unit to obtain the first target information. Then, the first target information is transmitted to the second recurrent network unit, so that the second recurrent network unit combines the first target information and the second modality feature to extract information to obtain the second target information, wherein the first target information and the second modality information are the hidden state information of different modality images respectively. Further, the hidden state information of each modality image is processed using a feedforward neural network model (MLP) to obtain a processing result, and then the processing results are fused to obtain fused information, and then the classification network is used to perform classification based on the fused information to obtain a classification result. The classification result includes a first probability that the object to be detected belongs to a living body and a second probability that it belongs to a non-living body, and then the final detection result is determined based on the first probability and the second probability. Exemplarily, if the first probability is greater than the second probability, it is determined that the object to be detected is a living body.

[0074] In the process of training the above-mentioned network units (at least one of the first recurrent network unit, the second recurrent network unit, the feature extraction unit and the MLP, etc.), the difference between the detection result of the sample detection object and the actual result can be used to adjust the network parameters of the network unit, thereby obtaining the trained network unit.

[0075] See also Figure 3 , Figure 3 1 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 30 includes a memory 31 and a processor 32 coupled to each other.

[0076] The memory 31 stores program instructions, and the processor 32 is used to execute the program instructions stored in the memory 31 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 30 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 30 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0077] Specifically, the processor 32 is used to control itself and the memory 31 to implement the steps of any of the above-mentioned embodiments. The processor 32 can also be called a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip with signal processing capabilities. The processor 32 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 32 can be implemented by an integrated circuit chip.

[0078] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 40 of the embodiment of the present application stores a program instruction 41, and when the program instruction 41 is executed, it implements the method provided by any embodiment of the above method and any non-conflicting combination. Among them, the program instruction 41 can form a program file and be stored in the above-mentioned computer-readable storage medium 40 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the methods of each implementation method of the present application. The aforementioned computer-readable storage medium 40 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0079] In the above scheme, after acquiring at least two modal images of the object to be detected, one of the modal images is used as the first modal image, and the remaining modal image is used as the second modal image, and then the first target information is first extracted from the first modal image, and then based on the first target information, the second target information is extracted from the second modal image, and then based on the fusion information of the first target information and the second target information, liveness detection is performed to obtain a liveness detection result. The present application guides the extraction of the second target information of the second modal image through the first target information of the first modal image, so that the extracted first target information and the second target information are complementary, and then the fusion information of the complementary first target information and the second target information is used for liveness detection. Compared with the method of directly extracting the first target information and the second target information and using their fusion information for liveness detection, the above method of the present application can effectively reduce the influence of irrelevant information on the liveness detection result, thereby improving the accuracy of liveness detection.

[0080] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0081] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for detecting a living body, characterized in that: The method comprises: Acquire at least two modality images of the object to be detected; Performing feature extraction on the first modality image to obtain first modality features; Extracting information from the first modal feature to obtain first target information; the first modal image is one of at least two modal images; Performing feature extraction on a second modality image to obtain a second modality feature; the second modality image is an image of at least two of the modality images excluding the first modality image; Using a target network to perform preset processing on the first target information and the second modal feature, and determining the retained data of the first target information and the impact data of the first target information on the second target information; the target network includes one of a second gated recurrent network unit or a long short-term memory network; Extracting the second target information from the first target information and the second modal features using the target network based on the second modal features, the retained data, and the influence data; the retained data is used to determine the degree of retention of the first target information in the second target information when extracting the second target information, and the influence data is used to determine data in the first target information and the second modal features that can be used as the second target information when extracting the second target information; Liveness detection is performed based on the fusion information of the first target information and the second target information to obtain a liveness detection result; the liveness detection result is used to indicate whether the object to be detected is alive.

2. The method according to claim 1, characterized in that The using the target network to perform preset processing on the first target information and the second modal feature to determine the retained data of the first target information and the impact data of the first target information on the second target information includes: splicing the first target information and the second modal feature to obtain spliced ​​data; Determine a first product of a first preset parameter and the spliced ​​data, and perform a first activation operation on the first product to obtain the retained data; Determine a second product of a second preset parameter and the spliced ​​data, and perform a second activation operation on the second product to obtain the influence data; wherein the first preset parameter and the second preset parameter are different learning parameters in the target network training process; The extracting the second target information from the first target information and the second modal feature using the target network based on the second modal feature, the retained data, and the influence data includes: determining a third product of the first target information and the impact data; Obtaining a concatenation result of the third product and the second modal feature, and performing a third activation operation on the concatenation result and a fourth product of a third preset parameter to obtain candidate information; The second target information is determined using the retained data, the first target information and the candidate information.

3. The method according to claim 2, characterized in that The determining the second target information by using the retained data, the first target information and the candidate information includes: Using the retention degree as a first weight parameter, and using the difference between the target constant and the retention degree as a second weight parameter; determining a fifth product of the first weight parameter and the first target information, and determining a sixth product of the second weight parameter and the candidate information; The sum of the fifth product and the sixth product is used as the second target information.

4. The method according to claim 1, characterized in that: At least two of the modal images are acquired by acquiring the object to be detected using at least two acquisition units, and the difference in image acquisition time of different acquisition units is less than a preset threshold; And / or, at least two of the modality images include at least two of a visible light image, a near infrared image, a depth map, and a thermal imaging image.

5. The method according to claim 1, characterized in that: The step of performing feature extraction on the first modality image to obtain the first modality feature is performed by a first feature extraction module, and the step of performing feature extraction on the second modality image to obtain the second modality feature is performed by a second modality feature extraction module. The first feature extraction module and the second modality feature extraction module are the same modules that share weights, or are different feature extraction modules.

6. The method according to claim 1, characterized in that The step of extracting information from the first modal feature is performed by a first gated recurrent network unit.

7. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 6.

Images