Feature recognition method and device, equipment and storage medium

Abstract

The invention provides a feature recognition method and device, equipment and a storage medium. The method comprises the steps of obtaining a feature vector corresponding to a target object; the feature vector is obtained through extraction after time synchronization and space synchronization are carried out on collected multi-modal data by a user terminal corresponding to the target object; the multi-modal data comprises image information, thermal imaging information, depth map information and time sequence video information; a cross-modal attention mechanism is adopted, and weight distribution is carried out on the feature vectors; and inputting the feature vector after weight allocation into a pre-trained recognition model to obtain a first recognition result corresponding to the target object. The method can shorten the cloud processing time and accelerate the response speed of the user terminal.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a feature recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the widespread application of facial recognition technology in fields such as finance and security, its typical feature recognition process heavily relies on cloud computing capabilities. Existing solutions typically involve terminal devices collecting user biometric data (such as facial images) and then uploading the raw data completely to a remote server via the network. The server, handling massive concurrent requests from numerous terminals, needs to invoke large-scale deep learning models for centralized feature extraction and comparison before finally returning the recognition results to the terminals.

[0003] This architecture has inherent bottlenecks, significantly slowing down the speed at which users obtain recognition results. On one hand, the entire "terminal-cloud-terminal" interaction process is constrained by network latency. More critically, the cloud server, as a centralized processing node, needs to respond to a large number of terminal requests simultaneously. In high-concurrency scenarios, computing resources are forced to be allocated among multiple tasks, and requests must queue for processing, easily creating processing bottlenecks and directly leading to extended average response times, failing to meet the extremely high real-time requirements of applications. Summary of the Invention

[0004] This application provides a feature recognition method, apparatus, device, and storage medium that can improve the speed of feature recognition.

[0005] In a first aspect, embodiments of this application provide a feature recognition method applied to a cloud-based interactive system, the interactive system including multiple user terminals, the cloud being communicatively connected to the user terminals, the method comprising:

[0006] The feature vector corresponding to the target object is obtained; the feature vector is extracted by the user terminal corresponding to the target object after performing time synchronization and spatial synchronization on the collected multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information;

[0007] A cross-modal attention mechanism is used to assign weights to the feature vectors;

[0008] The weighted feature vector is input into a pre-trained recognition model to obtain the first recognition result corresponding to the target object.

[0009] In one possible implementation, the method further includes:

[0010] After obtaining the trained recognition model, a pre-set generative model is used to generate virtual samples;

[0011] Based on the virtual samples, the recognition model is subjected to adversarial parameter tuning;

[0012] After knowledge distillation of the adversarial parameter-tuned recognition model, a lightweight recognition model is obtained and sent to each of the user terminals. This allows the user terminals to use a cross-modal attention mechanism to assign weights to the feature vectors after obtaining them, and then input the weighted feature vectors into the lightweight recognition model to obtain a second recognition result. Based on the first recognition result and the second recognition result, the user terminals obtain the feature recognition result corresponding to the target object.

[0013] In one possible implementation, the method further includes:

[0014] Continuously acquire the sample feature vector sent by the user terminal and the reference recognition result corresponding to the sample feature vector;

[0015] Based on the sample feature vector and the reference recognition result corresponding to the sample feature vector, the recognition model is continuously updated, and the updated recognition model is subjected to knowledge distillation to obtain an updated lightweight recognition model, which is then sent to each of the user terminals.

[0016] Secondly, embodiments of this application provide a feature recognition method applied to a user terminal of an interactive system, the interactive system including a cloud, the cloud being communicatively connected to the user terminal, the method comprising:

[0017] Obtain the multimodal data corresponding to the target object;

[0018] After performing time and spatial synchronization on the multimodal data, the feature vectors corresponding to the multimodal data are extracted; the multimodal data includes image information, thermal imaging information, depth map information, and temporal video information.

[0019] The multimodal data is sent to the cloud and the first recognition result is received; the first recognition result is obtained by the cloud using a cross-modal attention mechanism to weight the feature vector, and then inputting the weighted feature vector into a pre-trained recognition model.

[0020] A cross-modal attention mechanism is used to assign weights to the feature vectors;

[0021] The weighted feature vector is input into a pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object;

[0022] Based on the first recognition result and the second recognition result, the feature recognition result of the target object is determined.

[0023] In one possible implementation, after performing time and spatial synchronization on the multimodal data, extracting the feature vector corresponding to the multimodal data includes:

[0024] Based on the timestamps carried by the multimodal data, time synchronization is performed on each frame of the multimodal data;

[0025] Spatial transformation is performed on the time-synchronized multimodal data to achieve spatial synchronization;

[0026] Extract the feature vectors contained in the spatially synchronized multimodal data.

[0027] In one possible implementation, the method further includes:

[0028] Obtain the sample feature vector and the corresponding reference recognition result;

[0029] Based on the sample feature vector and the corresponding reference recognition result, the pre-stored lightweight recognition model is adjusted.

[0030] Thirdly, embodiments of this application provide a feature recognition device applied to the cloud of an interactive system, the interactive system including multiple user terminals, the cloud being communicatively connected to the user terminals, the device comprising:

[0031] The first acquisition module is used to acquire the feature vector corresponding to the target object; the feature vector is extracted by the user terminal corresponding to the target object after performing time synchronization and spatial synchronization on the collected multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information;

[0032] The allocation module is used to assign weights to the feature vectors using a cross-modal attention mechanism;

[0033] The first recognition module is used to input the weighted feature vector into a pre-trained recognition model to obtain the first recognition result corresponding to the target object.

[0034] Fourthly, embodiments of this application provide a feature recognition device applied to a user terminal of an interactive system, the interactive system including a cloud, the cloud being communicatively connected to the user terminal, the device comprising:

[0035] The second acquisition module is used to acquire the multimodal data corresponding to the target object;

[0036] An extraction module is used to extract the feature vectors corresponding to the multimodal data after performing time and spatial synchronization on the multimodal data; the multimodal data includes image information, thermal imaging information, depth map information, and temporal video information;

[0037] The receiving module is used to send the multimodal data to the cloud and receive the first recognition result; the first recognition result is obtained by the cloud using a cross-modal attention mechanism to weight the feature vector, and then inputting the weighted feature vector into a pre-trained recognition model;

[0038] The allocation module is used to assign weights to the feature vectors using a cross-modal attention mechanism;

[0039] The second recognition module is used to input the weighted feature vector into a pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object.

[0040] The determination module is used to obtain the feature recognition result of the target object based on the first recognition result and the second recognition result.

[0041] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0042] The memory stores instructions that the computer executes;

[0043] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect, the second aspect and / or various possible implementations of the second aspect as described above.

[0044] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect, the second aspect and / or various possible implementations of the second aspect.

[0045] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect, the second aspect and / or various possible implementations of the second aspect.

[0046] The feature recognition method, apparatus, device, and storage medium provided in this application, through a "cloud-user terminal" architecture, optimize the allocation of computational load, thereby achieving a qualitative leap in system response speed. By completely pre-processing the most time-consuming multimodal raw data and feature extraction steps, which were originally handled entirely by the cloud, the process is moved to the user terminal. Specifically, each user terminal utilizes its increasingly powerful computing capabilities to independently complete high-precision time synchronization, spatial alignment, and complex deep feature extraction of multimodal data. Ultimately, only a highly purified and standardized feature vector is uploaded to the cloud. The cloud no longer needs to perform repetitive and computationally intensive preprocessing and feature extraction of raw data for a massive number of user terminals. The simplified processing steps in the cloud greatly reduce the time it takes for the cloud to process a single request, allowing user terminals to obtain the first recognition result faster. This reduces delays caused by cloud resource competition and queuing, meeting the stringent requirements of instantaneous response in fintech scenarios. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 A schematic diagram of the interactive system provided in this application;

[0049] Figure 2 A flowchart illustrating the feature recognition method provided in this application;

[0050] Figure 3 A flowchart illustrating another feature recognition method provided in this application;

[0051] Figure 4 A schematic diagram of the feature recognition device provided in this application;

[0052] Figure 5 A schematic diagram of another feature recognition device provided in this application;

[0053] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] The feature recognition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, in the interactive system 100, the user terminal 102 can communicate with the cloud 104 via a network, and can also be directly or indirectly connected to the cloud 104 via wired communication.

[0057] Taking the application of feature recognition methods to the cloud 104 as an example, the financial institution's cloud 104 can obtain feature vectors corresponding to the target object from multiple user terminals 102 of the financial institution. These feature vectors are extracted by synchronizing the collected multimodal data in time and space through the user terminals 102 corresponding to the target object. The multimodal data includes image information, thermal imaging information, depth map information, and temporal video information. Furthermore, the cloud 104 can employ a cross-modal attention mechanism to assign weights to the feature vectors. The weighted feature vectors are then input into a pre-trained recognition model to obtain the first recognition result corresponding to the target object. The user terminals 102 can be, but are not limited to, devices deployed by the financial institution in various regions, such as various personal computers, laptops, smartphones, tablets, and IoT devices. The cloud 104 can be a cloud platform.

[0058] Taking the application of feature recognition method to user terminal 102 as an example, the user terminal 102 of the financial institution can extract the feature vectors corresponding to the multimodal data after performing time and space synchronization on the multimodal data. The multimodal data includes image information, thermal imaging information, depth map information, and time-series video information. Then, the multimodal data is sent to the cloud 104 of the financial institution and the first recognition result is received. The first recognition result is weighted by the feature vector through the cross-modal attention mechanism of the cloud 104, and the weighted feature vector is input into the pre-trained recognition model. The user terminal 102 further uses the cross-modal attention mechanism to weight the feature vector and inputs the weighted feature vector into the pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object. Finally, based on the first recognition result and the second recognition result, the feature recognition result of the target object is determined.

[0059] In one embodiment, a feature recognition method is provided. This embodiment uses the application of this feature recognition method to cloud 104 as an example for illustration. Figure 2 As shown, the feature recognition method includes:

[0060] Step 202: Obtain the feature vector corresponding to the target object; the feature vector is extracted by the user terminal corresponding to the target object after performing time synchronization and spatial synchronization on the collected multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information.

[0061] The target can refer to any user of a financial institution.

[0062] The user terminal may be equipped with at least one of the following: an RGB camera, an infrared sensor, a 3D structured light sensor, and a ToF sensor.

[0063] For example, the user terminal can acquire visible light images (i.e., image information) of the target object through an RGB camera on the user terminal, acquire infrared thermal images (i.e., thermal imaging information) of the target object through an infrared sensor on the user terminal, acquire 3D depth maps (i.e., depth map information) of the target object through a 3D structured light sensor on the user terminal, and acquire a short micro-expression time-series video of the target object (i.e., time-series video information) through a ToF sensor on the user terminal.

[0064] When the user terminal calls at least one of the RGB camera, infrared sensor, 3D structured light sensor, and ToF sensor, it sends a unified hardware trigger signal or instruction to all devices, commanding them to start collecting data at the same time. At this time, the timestamp information of all devices is taken from a high-precision system master clock of the user terminal, thereby ensuring that the time base of each modal data is consistent from the source, thus realizing the time synchronization of multimodal data.

[0065] Secondly, the user terminal can read the multimodal sensor parameters obtained in advance through the checkerboard calibration method from the storage module (such as flash memory), including: intrinsic parameter data: focal length (fx, fy), principal point coordinates (cx, cy), distortion coefficients (radial distortion k1 / k2, tangential distortion p1 / p2) of the RGB camera / thermal imager, and depth error compensation value of the depth camera; extrinsic parameter data: rotation matrix (R) and translation vector (T) of each sensor relative to the "terminal world coordinate system" - such as the extrinsic parameters R1 / T1 of the RGB camera, R2 / T2 of the thermal imager, and R3 / T3 of the depth camera. These parameters are the "conversion scale" for the user terminal to achieve coordinate unification.

[0066] The user terminal can further invoke a distortion correction algorithm to calculate the coordinates (u,v) of each pixel in the visible light image and infrared thermal image: based on the distortion coefficients in the loaded intrinsic parameters, using formulas (such as the radial distortion correction formula: u'=u+(u-cx)×(k1r)).2 +k2r 4 (where r is the distance from the pixel to the principal point), correcting the distorted pixel position to "distortion-free ideal pixel coordinates (u', v')", ensuring that the pixel positions of these two 2D images can accurately reflect the real spatial position.

[0067] Secondly, the user terminal can read the raw distance value d_raw of each pixel in the depth map, and combine it with the error compensation value of the depth camera (such as an error increase of 0.5cm for every 1 meter increase in distance) to calculate the calibrated true depth d = d_raw + (d_raw × error coefficient). At the same time, the user terminal can remove invalid values ​​in the depth map (such as 0 or maximum values ​​that are outside the measurement range) to avoid invalid data interfering with coordinate calculation, thereby achieving spatial synchronization.

[0068] In this step, the user terminal can, for example, feed visible light images into a lightweight CNN model (such as MobileNet) to extract detailed texture, contour, and detail features; feed infrared thermal images into a lightweight Transformer network to analyze unique thermal distribution patterns with its powerful global perception capabilities; feed 3D depth maps into a lightweight PointNet or 3D-CNN to directly process point cloud data and extract geometric structure features; and feed micro-expression videos of the target object into a lightweight LSTM or 3D-CNN to analyze the subtle dynamic movements of facial muscles between consecutive frames and obtain feature vectors.

[0069] Step 204: Use a cross-modal attention mechanism to assign weights to the feature vectors.

[0070] For example, in the cloud, the feature vectors can be input into a Transformer-based cross-modal attention module. The working principle of this module is to enable the interaction between feature vectors of different modalities and to automatically determine the reliability and importance of feature vectors of each modality in the current specific environment (such as dim lighting or abnormal temperature) by calculating attention weights.

[0071] For example, in well-lit environments, visible light images provide highly reliable texture features, and the attention mechanism assigns them a high weight. However, in backlit or low-light environments, the quality of visible light images is poor, and the mechanism automatically reduces their weight while correspondingly increasing the weight of features unaffected by visible light, such as infrared thermal imaging and 3D depth maps. This process is dynamic and adaptive, requiring no manual rule setting.

[0072] Step 206: Input the weighted feature vector into the pre-trained recognition model to obtain the first recognition result corresponding to the target object.

[0073] The aforementioned feature recognition method, through a "cloud-user terminal" architecture, optimizes the allocation of computational load, thereby achieving a qualitative leap in system response speed. By moving the most time-consuming multimodal raw data processing and feature extraction steps, originally handled entirely by the cloud, to the user terminal, each user terminal independently completes high-precision time synchronization, spatial alignment, and complex depth feature extraction of multimodal data (images, thermal imaging, depth maps, and time-series video) using its increasingly powerful computing capabilities (such as NPUs and GPUs). Ultimately, only a highly purified and standardized feature vector is uploaded to the cloud. The cloud no longer needs to perform repetitive and computationally intensive preprocessing and feature extraction of raw data for a massive number of user terminals. The simplified processing steps in the cloud significantly reduce the time it takes for the cloud to process a single request, allowing user terminals to obtain the first recognition result more quickly. This reduces latency caused by cloud resource competition and queuing, meeting the stringent requirements for instantaneous response in fintech scenarios.

[0074] In one embodiment, the method further includes:

[0075] After obtaining the trained recognition model, a pre-set generative model is used to generate virtual samples;

[0076] Adversarial parameter tuning of the recognition model is performed based on virtual samples;

[0077] After knowledge distillation of the adversarial parameter-tuned recognition model, a lightweight recognition model is obtained and sent to each user terminal. This enables the user terminal to use a cross-modal attention mechanism to assign weights to the feature vectors after obtaining them, and then input the weighted feature vectors into the lightweight recognition model to obtain a second recognition result. Based on the first and second recognition results, the feature recognition result corresponding to the target object is obtained.

[0078] In this embodiment, after obtaining the trained recognition model in the cloud, it is not deployed directly. Instead, an attack simulation phase is initiated. The cloud calls upon pre-set, advanced generative models (such as Diffusion Model or StyleGAN3), which are specifically designed to create highly realistic depth virtual attack samples. Examples include high-quality synthetic face images, virtual temperature distribution maps from infrared thermal imaging, and planar attack depth maps attempting to deceive 3D sensors. The generation process is diverse, capable of simulating different types of virtual samples under different lighting conditions and with different attack methods (such as screen capture, 3D masks, and advanced digital generation), thereby constructing a large and continuously updated library of fake samples.

[0079] Next, the cloud uses a large number of virtual samples as negative samples, along with a real sample library (containing multiple sample feature vectors and corresponding reference recognition results), to perform adversarial fine-tuning on the previously trained recognition model. This is a dynamic adversarial process: the generative model acts as the "spear," continuously generating new and more advanced attack samples; while the recognition model acts as the "shield," iteratively training on these virtual samples, continuously adjusting and updating its internal parameters, learning to recognize and resist subtle feature anomalies brought about by the latest virtual technologies. This process significantly improves the recognition model's generalization ability and robustness against unknown attacks.

[0080] While powerful recognition models achieved through adversarial parameter tuning are typically too complex and have too many parameters to run directly on resource-constrained user terminals, the cloud needs to employ knowledge distillation techniques. This large model is then used as a teacher model, where the judgment logic and decision boundaries of the input feature vectors are extracted and transferred to a student model with a simpler structure and fewer parameters. The resulting lightweight recognition model maintains high accuracy while being smaller and faster to compute.

[0081] Subsequently, the lightweight model was encrypted in the cloud and securely distributed to all user terminals for deployment and updates.

[0082] The second recognition result is obtained by the lightweight recognition model of the user terminal predicting the feature vector. When the first recognition result and the second recognition result are consistent and both have high confidence, the user terminal can directly output the feature recognition result.

[0083] If the confidence level of the local second identification result is low, or if it conflicts with the first identification result returned by the cloud, the user terminal can trigger an additional verification process (such as requiring the user to perform a specific action), or directly adopt the first identification result returned by the cloud as the final feature identification result.

[0084] The aforementioned feature recognition method, through a loop of generating virtual samples and adversarial training, enables the cloud to possess self-evolving model training capabilities. It can actively simulate and learn the latest attack methods, solving the problem of static models becoming outdated. This significantly enhances the sensitivity of the recognition model and improves overall recognition accuracy. Knowledge distillation technology allows the cloud to use large and complex recognition models to ensure the highest precision, while allowing user terminals to enjoy the low latency and high real-time experience brought by lightweight recognition models. Through the deployment of lightweight recognition models, even in the event of network interruption, user terminals can still provide a certain level of local security protection. Furthermore, by introducing a second local recognition result and comparing it with the first recognition result in the cloud, the reliability of the system is increased. The dual verification mechanism of the first and second recognition results makes the final feature recognition result more credible, providing higher security for interaction.

[0085] In one embodiment, the method further includes:

[0086] Continuously acquire sample feature vectors and corresponding reference recognition results sent by user terminals;

[0087] Based on the sample feature vector and the corresponding reference recognition result, the recognition model is continuously updated. After knowledge distillation, the updated lightweight recognition model is obtained and sent to each user terminal.

[0088] In this embodiment, the cloud continuously acquires sample feature vectors and reference recognition results for verified users from various user terminals, and uses this verified data to update and train the large-scale recognition model in the cloud. This ensures that model optimization does not require centralized storage of user data, fully protecting privacy. Furthermore, once the cloud-based recognition model has completed its iterative update, the system immediately initiates a knowledge distillation process to deploy the updated, more powerful, and lightweight recognition model to all user terminals, completing the continuous iterative upgrade of the model.

[0089] Based on the same inventive concept, such as Figure 3 As shown in the embodiments of this application, a feature recognition method is also provided, applied to a user terminal of an interactive system. The interactive system includes a cloud, and the cloud is communicatively connected to the user terminal. The method includes:

[0090] Step 302: Obtain the multimodal data corresponding to the target object.

[0091] Step 304: After synchronizing the multimodal data in time and space, extract the feature vectors corresponding to the multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information.

[0092] Step 306: Send the multimodal data to the cloud and receive the first recognition result; the first recognition result is obtained by using a cross-modal attention mechanism in the cloud to assign weights to the feature vectors, and then inputting the weighted feature vectors into the pre-trained recognition model.

[0093] Step 308: Use a cross-modal attention mechanism to assign weights to the feature vectors.

[0094] Step 310: Input the weighted feature vector into the pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object.

[0095] Step 312: Based on the first recognition result and the second recognition result, determine the feature recognition result of the target object.

[0096] In one embodiment, step 304 includes:

[0097] Based on the timestamps carried by the multimodal data, time synchronization is performed on each frame of the multimodal data;

[0098] Spatial transformation is performed on the time-synchronized multimodal data to achieve spatial synchronization;

[0099] Extract the feature vectors contained in the spatially synchronized multimodal data.

[0100] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0101] Based on the same inventive concept, this application also provides a feature recognition device for implementing the feature recognition method described above. The solution provided by this feature recognition device is similar to the solution described in the state determination method above. Therefore, the specific limitations in one or more device embodiments provided below can be found in the limitations of the feature recognition method above, and will not be repeated here.

[0102] In one embodiment, such as Figure 4As shown, a feature recognition device 400 is provided, applied to the cloud 104 of an interactive system 100. The interactive system 100 includes multiple user terminals 102, and the cloud 104 is communicatively connected to the user terminals 102. The device 400 includes:

[0103] The first acquisition module 402 is used to acquire the feature vector corresponding to the target object. The feature vector is extracted by the user terminal corresponding to the target object after performing time synchronization and spatial synchronization on the collected multimodal data. The multimodal data includes image information, thermal imaging information, depth map information and time-series video information.

[0104] The allocation module 404 is used to assign weights to the feature vectors using a cross-modal attention mechanism;

[0105] The first recognition module 406 is used to input the weighted feature vector into the pre-trained recognition model to obtain the first recognition result corresponding to the target object.

[0106] In one possible implementation, the first identification module 406 is further configured to:

[0107] After obtaining the trained recognition model, a pre-set generative model is used to generate virtual samples;

[0108] Adversarial parameter tuning of the recognition model is performed based on virtual samples;

[0109] After knowledge distillation of the adversarial parameter-tuned recognition model, a lightweight recognition model is obtained and sent to each user terminal. This enables the user terminal to use a cross-modal attention mechanism to assign weights to the feature vectors after obtaining them, and then input the weighted feature vectors into the lightweight recognition model to obtain a second recognition result. Based on the first and second recognition results, the feature recognition result corresponding to the target object is obtained.

[0110] In one possible implementation, the first identification module 406 is further configured to:

[0111] Continuously acquire sample feature vectors and corresponding reference recognition results sent by user terminals;

[0112] Based on the sample feature vector and the corresponding reference recognition result, the recognition model is continuously updated. After knowledge distillation, the updated lightweight recognition model is obtained and sent to each user terminal.

[0113] In one embodiment, such as Figure 5As shown, a feature recognition device 500 is provided, applied to the cloud 104 of an interactive system 100. The interactive system 100 includes multiple user terminals 102, and the cloud 104 is communicatively connected to the user terminals 102. The device 500 includes:

[0114] The second acquisition module 502 is used to acquire multimodal data corresponding to the target object;

[0115] The extraction module 504 is used to extract the feature vectors corresponding to the multimodal data after performing time and space synchronization on the multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information;

[0116] The receiving module 506 is used to send multimodal data to the cloud and receive the first recognition result; the first recognition result is obtained by the cloud using a cross-modal attention mechanism to assign weights to the feature vectors and then inputting the weighted feature vectors into a pre-trained recognition model.

[0117] The allocation module 508 is used to assign weights to the feature vectors using a cross-modal attention mechanism;

[0118] The second recognition module 510 is used to input the weighted feature vector into a pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object.

[0119] The determination module 512 is used to obtain the feature recognition result of the target object based on the first recognition result and the second recognition result.

[0120] In one possible implementation, the extraction module 504 is further configured to:

[0121] Based on the timestamps carried by the multimodal data, time synchronization is performed on each frame of the multimodal data;

[0122] Spatial transformation is performed on the time-synchronized multimodal data to achieve spatial synchronization;

[0123] Extract the feature vectors contained in the spatially synchronized multimodal data.

[0124] In one possible implementation, the second identification module 510 is further configured to:

[0125] Obtain the sample feature vector and the corresponding reference recognition result;

[0126] Based on the sample feature vector and the corresponding reference recognition result, adjust the pre-stored lightweight recognition model.

[0127] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0128] In the specific implementation process, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the above-mentioned feature recognition method.

[0129] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0130] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0131] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0132] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0133] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0134] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0135] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0136] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0137] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0140] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0142] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A feature recognition method, characterized in that, A cloud-based interactive system, comprising multiple user terminals, wherein the cloud is communicatively connected to the user terminals, the method comprising: The feature vector corresponding to the target object is obtained; the feature vector is extracted by the user terminal corresponding to the target object after performing time synchronization and spatial synchronization on the collected multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information; A cross-modal attention mechanism is used to assign weights to the feature vectors; The weighted feature vector is input into a pre-trained recognition model to obtain the first recognition result corresponding to the target object.

2. The method according to claim 1, characterized in that, The method further includes: After obtaining the trained recognition model, a pre-set generative model is used to generate virtual samples; Based on the virtual samples, the recognition model is subjected to adversarial parameter tuning; After knowledge distillation of the adversarial parameter-tuned recognition model, a lightweight recognition model is obtained and sent to each of the user terminals. This allows the user terminals to use a cross-modal attention mechanism to assign weights to the feature vectors after obtaining them, and then input the weighted feature vectors into the lightweight recognition model to obtain a second recognition result. Based on the first recognition result and the second recognition result, the user terminals obtain the feature recognition result corresponding to the target object.

3. The method according to claim 1, characterized in that, The method further includes: Continuously acquire the sample feature vector sent by the user terminal and the reference recognition result corresponding to the sample feature vector; Based on the sample feature vector and the reference recognition result corresponding to the sample feature vector, the recognition model is continuously updated, and the updated recognition model is subjected to knowledge distillation to obtain an updated lightweight recognition model, which is then sent to each of the user terminals.

4. A feature recognition method, characterized in that, A user terminal used in an interactive system, the interactive system including a cloud, the cloud being communicatively connected to the user terminal, the method comprising: Obtain the multimodal data corresponding to the target object; After performing time and spatial synchronization on the multimodal data, the feature vectors corresponding to the multimodal data are extracted; the multimodal data includes image information, thermal imaging information, depth map information, and temporal video information. The multimodal data is sent to the cloud, and the first recognition result generated by the cloud is received; the first recognition result is obtained by the cloud using a cross-modal attention mechanism to weight the feature vector, and then inputting the weighted feature vector into a pre-trained recognition model. A cross-modal attention mechanism is used to assign weights to the feature vectors; The weighted feature vector is input into a pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object; Based on the first recognition result and the second recognition result, the feature recognition result of the target object is determined.

5. The method according to claim 4, characterized in that, After performing time and spatial synchronization on the multimodal data, the feature vector corresponding to the multimodal data is extracted, including: Based on the timestamps carried by the multimodal data, time synchronization is performed on each frame of the multimodal data; Spatial transformation is performed on the time-synchronized multimodal data to achieve spatial synchronization; Extract the feature vectors contained in the spatially synchronized multimodal data.

6. The method according to claim 4, characterized in that, The method further includes: Obtain the sample feature vector and the corresponding reference recognition result; Based on the sample feature vector and the corresponding reference recognition result, the pre-stored lightweight recognition model is adjusted.

7. A feature recognition device, characterized in that, A cloud-based application for an interactive system, the interactive system including multiple user terminals, the cloud being communicatively connected to the user terminals, the device comprising: The first acquisition module is used to acquire the feature vector corresponding to the target object; the feature vector is extracted by the user terminal corresponding to the target object after performing time synchronization and spatial synchronization on the collected multimodal data; the multimodal data includes image information, thermal imaging information, depth map information and time-series video information; The allocation module is used to assign weights to the feature vectors using a cross-modal attention mechanism; The first recognition module is used to input the weighted feature vector into a pre-trained recognition model to obtain the first recognition result corresponding to the target object.

8. A feature recognition device, characterized in that, A user terminal for use in an interactive system, the interactive system including a cloud, the cloud being communicatively connected to the user terminal, the device comprising: The second acquisition module is used to acquire the multimodal data corresponding to the target object; An extraction module is used to extract the feature vectors corresponding to the multimodal data after performing time and spatial synchronization on the multimodal data; the multimodal data includes image information, thermal imaging information, depth map information, and temporal video information; The receiving module is used to send the multimodal data to the cloud and receive the first recognition result generated by the cloud; the first recognition result is obtained by the cloud using a cross-modal attention mechanism to assign weights to the feature vectors, and then inputting the weighted feature vectors into a pre-trained recognition model. The allocation module is used to assign weights to the feature vectors using a cross-modal attention mechanism; The second recognition module is used to input the weighted feature vector into a pre-stored lightweight recognition model to obtain the second recognition result corresponding to the target object. The determination module is used to obtain the feature recognition result of the target object based on the first recognition result and the second recognition result.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images