Behavioral Intent Recognition Methods and Devices, Training Methods, Media and Equipment

Abstract

This application provides a method and apparatus for behavioral intent recognition, a training method, a medium, and a device. The method includes: acquiring an image to be recognized; processing the image to be recognized based on a person feature subnet to obtain a person feature sequence; and processing the image to be recognized based on a device feature subnet to obtain a device feature sequence; fusing the person feature sequence and the device feature sequence to obtain a fused feature sequence; and processing the fused feature sequence using an intent recognition model to obtain an intent recognition result. This application uses a person / device subnet and a device feature subnet to process the image to be recognized, respectively. During the network training phase, the person feature subnet and the device feature subnet are trained independently, eliminating the need to collect joint samples of "human-held devices"; it can directly utilize publicly available human / hand datasets and device images collected online, reducing annotation costs.

Description

Technical Field

[0001] This application belongs to the field of computer vision processing technology, and relates to a method and apparatus for behavioral intent recognition, a training method, a medium and equipment. Background Technology

[0002] As the level of intelligence in the retail industry continues to improve, store inspection and payment behavior analysis are increasingly relying on computer vision technology. Among these technologies, accurately identifying payment-related behaviors, such as distinguishing between customers using their mobile phones to display payment codes and store clerks using barcode scanners to collect payments, is of key value for optimizing operational processes, ensuring transaction security, and analyzing consumer habits.

[0003] Currently, mainstream solutions typically follow a phased perception process. First, object detection models (such as YOLO, Faster R-CNN, etc.) are used to detect human bodies in the input video frames or images, locating and cropping out each human body region (called a "person frame") in the image. Then, for each cropped human body region, a fine-grained classification model or attribute recognition model is further used to determine whether the person is holding a specific device, such as "holding a mobile phone" or "holding a barcode scanner". Finally, based on the attribute recognition results, combined with preset rules (e.g., the person holding a barcode scanner is usually a store clerk, and the person holding a mobile phone may be a customer), the identity of the person (customer / store clerk) and their ongoing behavior (payment / receipt) are inferred.

[0004] Although the above solutions have been applied to some extent, they have several inherent and interrelated technical limitations that severely restrict their accuracy, robustness, and practicality in real-world, complex retail environments:

[0005] Sample collection is difficult, and the model relies on strongly coupled labeled data: The core of existing methods lies in training an attribute model that can accurately identify the joint state of "human-device". This requires a large-scale, high-quality training dataset, which must contain a large number of precisely labeled image samples of "humans holding a specific model of mobile phone" and "humans holding a specific model of barcode scanner". In practical applications, the collection of such combined samples is extremely costly and time-consuming, and it is difficult to cover all possible device models, holding postures, and environmental backgrounds, making data scarcity the primary bottleneck for technology implementation.

[0006] Poor generalization ability and difficulty adapting to changes in devices and scenarios: Because the model learns the strong correlation between "people" and "specific appearance devices," its recognition ability is highly dependent on the device styles in the training data. When new models of mobile phones, barcode scanners of different colors or shapes appear, or even other electronic devices (such as tablets) not in the training set appear, the model's recognition accuracy will drop significantly. This requires the system to recollect data and retrain the model when encountering new devices, making it impossible to achieve effective zero-shot or few-shot generalization, and resulting in high maintenance and update costs.

[0007] The behavioral representations are coarse, and the ability to understand intent is insufficient: Existing technologies typically only output binary or coarse-grained classification results such as "whether a device is being held," lacking the ability to characterize more subtle behavioral intentions. For example, it cannot effectively distinguish between a customer's payment behavior of "actively displaying their mobile payment code" and the unrelated behavior of "merely holding their phone"; similarly, it is difficult to determine whether a store clerk is "using a barcode scanner" or "simply holding the barcode scanner in their hand." This coarse-grained output limits the depth and value of behavioral analysis.

[0008] Low interpretability and credibility: Existing solutions typically operate as an end-to-end classification "black box," ultimately providing only a behavior category label. They lack supporting visual evidence or logical reasoning processes for the decision, such as "the decision to use QR code payment was based on the detection of specific features of the barcode scanner in the human hand area, and the spatial pointing relationship between the device and the screen in another human body area." This lack of interpretability makes it difficult for the system to undergo manual review and quickly diagnose the root cause of errors when misjudgments occur, thus reducing its credibility in critical security or auditing scenarios. Summary of the Invention

[0009] In view of the above-mentioned technical problems, this application provides a method and apparatus for recognizing behavioral intentions, a training method, a medium and a device.

[0010] To achieve the above and other related objectives, a first aspect of this application provides a method for recognizing behavioral intent. The method includes: acquiring an image to be recognized;

[0011] The image to be identified is processed based on a person feature subnet to obtain a person feature sequence; and

[0012] The image to be identified is processed based on the device feature subnet to obtain a device feature sequence;

[0013] The character feature sequence and the device feature sequence are fused to obtain a fused feature sequence;

[0014] The fused feature sequence is processed using an intent recognition model to obtain intent recognition results.

[0015] In some embodiments of the first aspect of this application, the image to be identified is processed based on a person feature subnet to obtain a person feature sequence, including:

[0016] Human body regions are detected and cropped from the image to be identified;

[0017] The human body region image is used to extract features through the backbone network to obtain an initial human body feature map;

[0018] The initial human feature map is processed by a key point prediction network to obtain several key point heatmaps, and key point feature sequences and orientation feature sequences are obtained based on the several key point heatmaps.

[0019] The initial human feature map is segmented into hand regions using a segmentation network to obtain hand region images, and a gesture feature sequence is obtained based on the hand region images.

[0020] The key point feature sequence, the direction feature sequence, and the gesture feature sequence are concatenated to obtain the character feature sequence.

[0021] In some embodiments of the first aspect of this application, key point feature sequences and orientation feature sequences are obtained based on several key point heatmaps, including:

[0022] The soft maximum position and variance of several key point heatmaps are calculated respectively to obtain the sub-pixel coordinate set and confidence score set of the key points;

[0023] The sub-pixel coordinate set is weighted or filtered based on the confidence score set to obtain an optimized key point data set; and wrist and elbow coordinates are filtered from the sub-pixel coordinate set, and forearm direction vector is calculated based on the wrist and elbow coordinates.

[0024] The key point data set is mapped using a multilayer perceptron to obtain a key point feature sequence; and the forearm direction vector is mapped using a multilayer perceptron to obtain a direction feature sequence.

[0025] In some embodiments of the first aspect of this application, the image to be identified is processed based on a device feature subnet to obtain a device feature sequence, including:

[0026] Detect and crop out the device area image from the image to be identified;

[0027] The device region image is used to extract features through the backbone network to obtain the initial feature map of the device;

[0028] The initial feature map of the device is segmented and statistically analyzed using a segmentation network to obtain a component probability map;

[0029] Based on the component probability map, the component feature sequence and main axis direction sequence of each component are obtained respectively;

[0030] The initial feature map of the device is classified using a classifier to obtain the device category;

[0031] The component feature sequence, the spindle direction sequence, and the equipment category are spliced ​​together to obtain the equipment feature sequence.

[0032] In some embodiments of the first aspect of this application, the person feature sequence and the device feature sequence are fused to obtain a fused feature sequence, including:

[0033] Cross-attention learning is performed on the person feature sequence, where the person feature sequence is used as the query and the device feature sequence is used as the key and value, to obtain the updated person feature sequence.

[0034] Cross-attention learning is performed on the device feature sequence, where the device feature sequence is used as the query and the person feature sequence is used as the key and value, to obtain the updated device feature sequence.

[0035] The updated character feature sequence and the updated device feature sequence are concatenated to obtain a concatenated feature sequence;

[0036] The spliced ​​feature sequence is input into an attention encoder for further encoding to obtain a fused feature sequence.

[0037] In some embodiments of the first aspect of this application, the intent recognition result includes: intent classification result and alignment regression value. The fused feature sequence is processed using an intent recognition model to obtain the intent recognition result, including:

[0038] The fused feature sequence is pooled to obtain a global feature vector;

[0039] The global feature vector is input into the classification subnetwork and the regression subnetwork, respectively;

[0040] The classification subnetwork outputs the intention classification result, and the regression subnetwork outputs the alignment regression value.

[0041] In some embodiments of the first aspect of this application, processing the fused feature sequence using an intent recognition model to obtain an intent recognition result further includes:

[0042] Obtain the geometric feature sequence;

[0043] The global feature vector is fused with the geometric feature sequence to obtain the regression input features;

[0044] The regression input features are input into the regression sub-network to obtain the angle between the device and the person's hand.

[0045] To achieve the above and other related objectives, a second aspect of this application provides a method for training an intent recognition model. The training method includes:

[0046] Separately acquire a people dataset and a device dataset; wherein, the device dataset is a hybrid dataset, including device images obtained from the network and real-world images;

[0047] A person feature subnet is trained based on the person dataset, and a device feature subnet is trained based on the device dataset.

[0048] Using the trained person feature subnet and the trained device feature subnet, the person dataset and the device dataset are processed offline to generate person feature sequence dataset and device feature sequence dataset, respectively.

[0049] The intent recognition model is trained based on the person feature sequence dataset and the device feature sequence dataset;

[0050] A set of images to be identified in a real scene is obtained, and the set of images to be identified is processed using the person feature subnet and the device feature subnet respectively to obtain paired data of real person dataset and real device dataset;

[0051] The intent recognition model is fine-tuned based on the pairing data; wherein at least some parameters of the person feature subnet and the device feature subnet are frozen.

[0052] To achieve the above and other related objectives, a third aspect of this application provides a behavioral intent recognition device. The device includes:

[0053] The acquisition module is configured to acquire the image to be recognized.

[0054] The human feature extraction module is configured to process the image to be identified based on a human feature subnet to obtain a human feature sequence;

[0055] The device feature extraction module is configured to process the image to be identified based on the device feature subnet to obtain a device feature sequence;

[0056] The feature fusion module is configured to fuse the person feature sequence and the device feature sequence to obtain a fused feature sequence;

[0057] The intent recognition module is configured to process the fused feature sequence using an intent recognition model to obtain an intent recognition result.

[0058] To achieve the above and other related objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the behavioral intent recognition method of any one of the first aspects and / or the intent recognition model training method of any one of the second aspects.

[0059] To achieve the above and other related objectives, the fifth aspect of this application provides an electronic device, including a memory and a processor;

[0060] The memory is configured to store computer programs;

[0061] The processor is configured to execute a computer program stored in the memory, so that the electronic device, when executed, implements the behavioral intent recognition method of any one of the first aspects and / or the intent recognition model training method of any one of the second aspects.

[0062] As described above, the behavioral intent recognition method, apparatus, training method, medium, and equipment described in this application have the following beneficial effects:

[0063] This application decouples the overall recognition process into independent "person feature subnet" and "device feature subnet" for separate training, eliminating the reliance on scarce "person-device" joint annotation data found in traditional solutions. The person network can be trained using publicly available and abundant human or hand datasets; the device network only requires images of the device itself, which can be obtained at low cost through web scraping. This design frees model construction from the constraints of specific combinations of samples that are difficult to collect, greatly reducing data annotation costs and the project startup threshold. When new devices appear that are not present in the training set, only the device feature subnet needs to be updated, and the intent recognition model can achieve generalized recognition even without joint samples.

[0064] When it is necessary to identify new payment devices (such as new barcode scanners, smartwatches, and other devices including but not limited to payment terminals, wearable devices, and smart interactive devices), only incremental training or model updates of the independent device feature subnet are required. There is no need to re-collect and label the combined data of the device and the person, nor is it necessary to modify or retrain the person feature subnet. This modular design gives the system efficient iteration and expansion capabilities, enabling it to quickly adapt to the upgrading of retail terminal devices and significantly reduce the long-term maintenance costs of the system.

[0065] Because the person feature subnet and the device feature subnet are trained separately, overfitting to specific "person-device" combination patterns in the training set is effectively avoided. The final determination of payment behavior intent is based on reasoning based on high-level semantic features (Tokens) and their spatial and temporal relationships, rather than directly recognizing low-pixel-level combined images. This enables the system to maintain stable recognition capabilities for payment devices of different brands, models, and appearances, fundamentally improving the model's generalization and robustness in real and complex scenarios.

[0066] It can analyze deeper behavioral intentions. This intention-level attribute recognition makes the behavioral analysis results more closely aligned with the needs of retail business scenarios (such as payment verification and process compliance checks), greatly improving the practical value of the analysis results.

[0067] By integrating human and device feature sequences and outputting visual information such as attention weights, the key visual evidence on which the final decision is based is clearly indicated. This not only enhances the credibility of the results but also provides a clear and direct basis for manual review, diagnosis of misjudgment cases, and model optimization. Attached Figure Description

[0068] Figure 1 The diagram shown illustrates one implementation scenario of the behavioral intent recognition method described in this application.

[0069] Figure 2a The diagram shown is a flowchart illustrating the behavioral intent recognition method described in the embodiments of this application.

[0070] Figure 2b The diagram shown is a flowchart illustrating a behavioral intent recognition method according to another embodiment of this application.

[0071] Figure 3 The diagram shown is a flowchart of the workflow of the person feature subnet described in this application embodiment.

[0072] Figure 4a The diagram shows a flowchart of obtaining a person's feature sequence as described in an embodiment of this application.

[0073] Figure 4b The diagram shown is a flowchart illustrating the process of obtaining a person's feature sequence as described in another embodiment of this application.

[0074] Figure 5 The diagram shows a flowchart of obtaining device feature sequences as described in an embodiment of this application.

[0075] Figure 6 The diagram shows a flowchart of obtaining the fusion feature sequence as described in an embodiment of this application.

[0076] Figure 7The diagram shows a flowchart of obtaining intent recognition results as described in an embodiment of this application.

[0077] Figure 8a The diagram shown is a flowchart illustrating the intent recognition model training method described in an embodiment of this application.

[0078] Figure 8b The diagram shown is a flowchart illustrating the intent recognition model training method according to another embodiment of this application.

[0079] Figure 9 The diagram shown is a structural schematic of the behavioral intent recognition device described in the embodiments of this application.

[0080] Figure 10 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0081] Component designation explanation

[0082] 11 cell phone 12 Tablet PC 13 laptop 90 Behavioral Intent Recognition Device 91 Get Module 92 Character Feature Extraction Module 93 Device Feature Extraction Module 94 Feature fusion module 95 Intent recognition module 10 electronic devices 101 processor 102 memory 1021 operating system 1022 app 103 Network interface 104 bus system 105 User Interface Detailed Implementation

[0083] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0084] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0085] The behavioral intent recognition method of this application can be applied to, for example... Figure 1The electronic devices shown in this application may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The specific types of electronic devices are not limited in this application embodiment.

[0086] For example, electronic devices can communicate with networks and other devices wirelessly. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0087] The principles and implementation methods of the behavioral intent recognition method and apparatus, training method, medium and device of this application will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can understand the behavioral intent recognition method and apparatus, training method, medium and device of this embodiment without creative effort.

[0088] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 2a and Figure 2b Detailed explanation. For example... Figure 2a and Figure 2b As shown, the behavioral intent recognition method of this application embodiment includes the following steps S21 to S25.

[0089] Step S21: Obtain the image to be recognized.

[0090] Step S22: Process the image to be identified based on the human feature subnet to obtain the human feature sequence.

[0091] Step S23: Process the image to be identified based on the device feature subnet to obtain the device feature sequence.

[0092] Step S24: Merge the person feature sequence and the device feature sequence to obtain a fused feature sequence.

[0093] Step S25: Process the fused feature sequence using the intent recognition model to obtain the intent recognition result.

[0094] Specifically, the image to be identified is a color image. ,in, For the height of the image to be identified, The width of the image to be recognized.

[0095] The following embodiments will describe the process of step S22, which involves processing the image to be recognized based on the human feature subnet to obtain the human feature sequence.

[0096] In this embodiment, a person detection box is extracted from the image to be identified, and the extracted person detection box is preprocessed to obtain a human body region image. Then, the human body region image is input into the person feature subnet for feature extraction and feature stitching to obtain a person feature sequence.

[0097] In this embodiment, the person feature subnetwork includes: a backbone network layer, a key point prediction network layer, a segmentation network layer, and a splicing output layer, wherein the key point prediction network layer shares the output of the backbone network layer with the segmentation network layer.

[0098] In some implementations, such as Figure 3 As shown, the human feature subnetwork is used to process the image to be recognized to obtain a human feature sequence. This process includes: detecting and cropping the human body region image from the image to be recognized; extracting features from the human body region image through the backbone network of the human feature subnetwork to obtain an initial human body feature map; processing the initial human body feature map through the key point prediction network of the human feature subnetwork to obtain several key point heatmaps, and further obtaining key point feature sequences and orientation feature sequences; segmenting the initial human body feature map into hand regions through the segmentation network of the human feature subnetwork to obtain hand region images, and further obtaining gesture feature sequences; and concatenating the obtained key point feature sequences, orientation feature sequences, and gesture feature sequences to obtain the human feature sequence.

[0099] like Figure 4a and Figure 4b As shown, obtaining the character feature sequence includes the following steps S41 to S45.

[0100] Step S41: Detect and crop out the human body region image from the image to be identified.

[0101] Step S42: Extract features from the human body region image through the backbone network to obtain the initial human body feature map.

[0102] Step S43: The initial human feature map is processed by the key point prediction network to obtain several key point heatmaps, and the key point feature sequence and orientation feature sequence are obtained based on the several key point heatmaps.

[0103] Step S44: The initial human feature map is segmented into hand regions using a segmentation network to obtain hand region images, and a gesture feature sequence is obtained based on the hand region images.

[0104] Step S45: Concatenate the key point feature sequence, direction feature sequence, and gesture feature sequence to obtain the character feature sequence.

[0105] In this embodiment, target detectors such as YOLO and DETR are used to identify the image. To perform the detection, a human detection frame is obtained. The detection frame of the obtained person Perform image cropping of people and then use the bounding boxes of the cropped people. The image is scaled to a fixed size and then normalized to obtain the human body region image. .

[0106] In this embodiment, the human body region image is... The image of the human body region is input into the backbone network. Perform forward computation on the network to obtain the initial human feature map. ,in , Initial feature map of the human body The number of channels, Initial feature map of the human body height, Initial feature map of the human body The width.

[0107] In this embodiment, the backbone network can be a network structure such as VIT-S / 16 or HRNet-W32.

[0108] In this embodiment, the obtained initial human feature map Input into the keypoint prediction network to predict Heatmap of key points ,in, , For the number of people, This represents the number of key point heatmaps.

[0109] In this embodiment, key point feature sequences and orientation feature sequences are obtained based on several key point heatmaps, including:

[0110] The soft maximum position and variance of several keypoint heatmaps are calculated to obtain the sub-pixel coordinate set and confidence score set of the keypoints. The sub-pixel coordinate set is then weighted or filtered based on the confidence score set to obtain an optimized keypoint data set. Wrist and elbow coordinates are then filtered from the sub-pixel coordinate set, and forearm direction vectors are calculated based on these coordinates. The keypoint data set is then mapped using a multilayer perceptron to obtain a keypoint feature sequence. Finally, the forearm direction vector is mapped using a multilayer perceptron to obtain a direction feature sequence.

[0111] Specifically, regarding the acquired key point heatmap Heatmap of each key point Treating it as a two-dimensional probability distribution, a weighted average is taken of all pixels to obtain a heatmap of key points. The corresponding precise and continuous two-dimensional coordinates of key points, i.e., the set of sub-pixel coordinates of key points;

[0112] Heatmap of key points Variance estimation is performed on all pixels to obtain the confidence score of each pixel, i.e., the set of confidence scores. The obtained sub-pixel coordinate set is weighted or filtered based on the set of confidence scores to obtain an optimized key point data set. The key point data set includes the normalized coordinates and confidence scores of the key points. The key point data set is mapped through a multilayer perceptron (MLP) to obtain the key point feature sequence.

[0113] The formula for mapping the key point data set using a multilayer perceptron (MLP) is as follows:

[0114]

[0115] in, Key point Normalized coordinates, Key point The confidence score, Indicates dimension.

[0116] Obtain elbow keypoint coordinates from subpixel coordinate set Coordinates of key points on the wrist Calculate the coordinates of key points at the elbow. Coordinates of wrist key points The difference between them is used to obtain the original forearm direction vector. Multilayer perceptron (MLP) was used to analyze the original orientation vector of the forearm. A mapping process is performed to obtain a directional feature sequence.

[0117] The formula for calculating the original direction vector of the forearm is:

[0118]

[0119] in, The original direction vector of the forearm. The x-axis coordinates of key points on the wrist. Here are the x-coordinates of the key points at the elbow. The y-axis coordinates of key points on the wrist. The y-axis coordinates of the key points at the elbow are given.

[0120] The formula for mapping the original forearm orientation vector using a multilayer perceptron (MLP) is as follows:

[0121]

[0122] in, Elbow key point coordinates Coordinates of key points on the wrist The angle between them Indicates dimension.

[0123] In this embodiment, the hand region of the initial human feature map is located, according to... The requirement is to take a square area centered on the wrist key point of the hand region to obtain the hand region image;

[0124] The acquired hand region image is sequentially processed with alignment, scaling, global average pooling, and multilayer perceptron (MLP) mapping operations to obtain the gesture feature sequence, i.e.

[0125]

[0126] in, This is an initial feature map of the human body. Indicates dimension.

[0127] The following embodiments will describe the process of step S23, which involves processing the image to be identified based on the device feature subnet to obtain the device feature sequence.

[0128] In this embodiment, the device detection box in the image to be identified is extracted, and the extracted device detection box is preprocessed to obtain the device region image. Then, the device region image is input into the device feature subnet for feature extraction to obtain the device feature sequence.

[0129] In some implementations, processing the image to be identified based on a device feature subnetwork to obtain a device feature sequence includes: detecting and cropping the device region image from the image to be identified; extracting features from the device region image using a backbone network to obtain an initial device feature map; segmenting and statistically analyzing the initial device feature map using a segmentation network to obtain a component probability map; weighting the initial device feature map using the component probability map to obtain a component feature sequence for each component; binarizing the component probability map and calculating the covariance matrix of the device in the initial device feature map based on the binarized component probability map; obtaining the principal axis direction vector of the device based on the covariance matrix and further obtaining the principal axis direction sequence of the device; classifying the initial device feature map using a classifier to obtain the device category; and concatenating the component feature sequence, principal axis direction sequence, and device category to obtain the device feature sequence.

[0130] like Figure 5 As shown, obtaining the device feature sequence includes the following steps S51 to S56:

[0131] Step S51: Detect and crop the device area image from the image to be identified.

[0132] Step S52: Extract features from the device region image through the backbone network to obtain the initial feature map of the device.

[0133] Step S53: Use a segmentation network to segment and statistically analyze the initial feature map of the device to obtain a component probability map.

[0134] Step S54: Based on the component probability map, obtain the component feature sequence and main axis direction sequence of each component.

[0135] Step S55: Use a classifier to classify the initial feature map of the device to obtain the device category.

[0136] Step S56: Assemble the component feature sequence, spindle direction sequence, and equipment category to obtain the equipment feature sequence.

[0137] In this embodiment, target detectors such as YOLO and DETR are used to identify the image. Perform testing to obtain the equipment testing frame. The obtained equipment detection frame Perform device image cropping and then select the cropped device detection frame. The image is scaled to a fixed size and then normalized to obtain the device area image. .

[0138] In this embodiment, the device area image is... Input to the backbone network for device area images Perform network forward processing to obtain the initial feature map of the device. ,in , Initial feature map of the device The number of channels, Initial feature map of the device height, Initial feature map of the device The width.

[0139] In this embodiment, the backbone network can be a network structure such as VIT-S / 16 or HRNet-W32.

[0140] The initial feature map of the device is processed by the segmenter. Segmentation is performed to obtain the component probability map. ,in, This indicates the number of each component, with the total number of different components equal to 1. Optionally, the segmenter can be implemented using dilated convolution or a lightweight mask head.

[0141] In this embodiment, the initial feature map of the device is weighted and pooled using the component probability map to obtain the component feature sequence of each component.

[0142] Specifically, to ensure that each component is in accordance with the initial design equipment feature diagram The feature information in the preliminary equipment feature diagram Use the obtained component probability map Weighted pooling is performed, and the results of weighted pooling are mapped using a multilayer perceptron (MLP) to obtain the component feature sequence of each component.

[0143] Among them, the preliminary design equipment feature diagram Use the obtained component probability map The formula for weighted pooling is:

[0144]

[0145]

[0146] in, Represents coordinates.

[0147] In this embodiment, the component probability map is binarized, and the covariance matrix of each component in the initial feature map of the device is calculated based on the binarized result. The principal axis direction vector of the device is obtained based on the covariance matrix, and the principal axis direction sequence of the device is further obtained.

[0148] In this embodiment, the initial feature map of the device is processed by a classifier. The system performs classification and outputs classification results, which include component type and component probability. Component type includes at least one or more of the following: barcode scanner, mobile phone, POS machine, and others. Optionally, the classifier is implemented using a global average pooling layer and a fully connected layer.

[0149] The device categories are mapped using a multilayer perceptron (MLP), and the mapped device categories are then concatenated with the unseen feature sequence and the main axis direction sequence.

[0150] In some implementations, the fusion of person feature sequences and device feature sequences to obtain a fused feature sequence includes:

[0151] like Figure 6 As shown, obtaining the fused feature sequence includes the following steps S61 to S64:

[0152] Step S61: Perform cross-attention learning on the person feature sequence, where the person feature sequence is used as the query and the device feature sequence is used as the key and value, to obtain the updated person feature sequence.

[0153] Step S62: Perform cross-attention learning on the device feature sequence, where the device feature sequence is used as the query and the person feature sequence is used as the key and value, to obtain the updated device feature sequence.

[0154] Step S63: The updated character feature sequence and the updated device feature sequence are concatenated to obtain a concatenated feature sequence.

[0155] Step S64: Input the spliced ​​feature sequence into the attention encoder for further encoding to obtain the fused feature sequence.

[0156] In this embodiment, a person feature sequence is used as the primary query parameter, and a device feature sequence is used as the key and value. The learning emphasizes how people use devices and allows for bidirectional interaction between people and devices.

[0157] in,

[0158]

[0159] in, Represents a sequence of character traits. Represents the device characteristic sequence. The query projection matrix represents the sequence of character features. The key projection matrix represents the sequence of device features. The value projection matrix represents the sequence of device features. This indicates the similarity between person feature sequences and device feature sequences. The position offset, representing the relative position, is achieved through... Encourage space providers to have more relevant people and equipment. This is the updated sequence of character traits.

[0160] In this embodiment, device feature sequences are used as the primary query method, and person feature sequences are used as keys and values.

[0161] in,

[0162] in, The query projection matrix represents the sequence of device features. The key projection matrix represents the sequence of human characteristics. The value projection matrix represents the sequence of human features. This is the updated device feature sequence.

[0163] In this embodiment, the spliced ​​feature sequence is represented as:

[0164] in, For the updated character feature sequence, This is the updated device feature sequence.

[0165] In this embodiment, the attention encoder includes several transformer layers, each of which consists of a feedforward neural network layer, a residual connection layer, and a normalization layer.

[0166] splicing feature sequences The signal is further encoded by an attention encoder, resulting in a stack of several cross-attention layers. Each layer includes a bidirectional interaction between the person's feature sequence and the device's feature sequence, with positional bias. Using anchor point coordinate difference As input, it is processed through a multilayer perceptron (MLP) mapping to obtain a fused feature sequence. .

[0167] The following embodiments will describe the implementation process of step S25, which uses an intent recognition model to process the fused feature sequence to obtain the intent recognition result.

[0168] Specifically, the fused feature sequences As input, the fused feature sequence is processed by a trained intent recognition model. The process is performed to obtain the intent recognition result.

[0169] The intent recognition results include: intent classification results and alignment regression values; intent classification results include: no operation, handheld barcode scanner, device idle, etc., and alignment regression values ​​are... The values ​​between these values ​​are used to determine whether the device is pointing at the corresponding person. The alignment regression value is the geometric relationship between the device's principal axis direction vector and the person's forearm's original direction vector. The geometric relationship includes the included angle value, normalized similarity, or a combination thereof.

[0170] In this embodiment, the intention recognition model is used to process the fused feature sequence to obtain the intention recognition result, including: performing a pooling operation on the fused feature sequence to obtain a global feature vector; inputting the global feature vector into the classification subnetwork and the regression subnetwork respectively; outputting the intention classification result through the classification subnetwork and outputting the alignment regression value through the regression subnetwork.

[0171] like Figure 7 As shown, obtaining the intent recognition result includes the following steps S71 to S73:

[0172] Step S71: Perform pooling operation on the fused feature sequence to obtain the global feature vector.

[0173] Step S72: Input the global feature vector into the classification subnetwork and the regression subnetwork respectively.

[0174] Step S73: Output the intent classification result through the classification subnetwork and output the alignment regression value through the regression subnetwork.

[0175] In some implementations, obtaining the intent recognition result further includes the following steps S74 to S76:

[0176] Step S74: Obtain the geometric feature sequence.

[0177] Step S75: The global feature vector and the geometric feature vector are fused to obtain the regression input features.

[0178] Step S76: Input the regression input features into the regression subnetwork to obtain the angle between the device and the person's hand.

[0179] In this embodiment, the geometric feature sequence represents the positional relationship between people and devices on the image.

[0180] In some implementations, the center point of the human body in the image to be identified is selected. and equipment center point Human body center point and equipment center point Calculate the relative displacement, direction difference, and scale ratio between the two.

[0181] The center point of the human body is determined by a multilayer perceptron (MLP). and equipment center point The relative displacements, orientation differences, and scale ratios between the components are mapped to obtain a geometric feature sequence.

[0182] Human body center point and equipment center point The formula for calculating the relative displacement between them is:

[0183]

[0184] Human body center point and equipment center point The formula for calculating the directional difference between them is:

[0185]

[0186]

[0187] Human body center point and equipment center point The formula for calculating the scale ratio between them is:

[0188] .

[0189] in, The original direction vector of the forearm. This is the device's original direction vector.

[0190] This application also provides a method for training an intent recognition model.

[0191] like Figure 8a and Figure 8b The diagram shows an intent recognition model training method according to an embodiment of this application, including the following steps S81 to S86:

[0192] Step S81: Obtain the person dataset and the device dataset respectively; wherein, the device dataset is a hybrid dataset, including device images obtained from the network and real-world images.

[0193] Step S82: Train a human feature subnet based on the human dataset and a device feature subnet based on the device dataset.

[0194] Step S83: Using the trained person feature subnet and the trained device feature subnet, the person dataset and device dataset are processed offline to generate person feature sequence dataset and device feature sequence dataset, respectively.

[0195] Step S84: Train the intent recognition model based on the person feature sequence dataset and the device feature sequence dataset.

[0196] Step S85: Obtain the set of images to be identified in the real scene, and process the set of images to be identified using the person feature subnet and the device feature subnet respectively, so as to obtain the paired data of the real person dataset and the real device dataset.

[0197] Step S86: Fine-tune the intent recognition model based on the paired data; wherein at least some parameters of the person feature subnet and the device feature subnet are frozen.

[0198] In this embodiment, the human feature subnetwork includes: a backbone network layer, a key point prediction network layer, a segmentation network layer, and a splicing output layer.

[0199] The loss function for the character feature subnet is designed as follows:

[0200] in, Predicting losses for key points, For hand part cutting loss, Loss in the forearm direction.

[0201] The keypoint prediction loss function of the keypoint prediction network layer is designed as follows:

[0202]

[0203] in, Represents the true heatmap of key points. This represents a heatmap for predicting key points.

[0204] The keypoint prediction network layer also includes a forearm orientation calculation layer, whose forearm orientation loss function is designed as follows:

[0205]

[0206] in, This represents the actual forearm direction vector. This represents the predicted forearm direction vector.

[0207] The hand-segmentation loss function for segmenting network layers is designed as follows:

[0208]

[0209] in, Represents the actual gesture feature sequence, This represents the predicted gesture feature sequence.

[0210] In this embodiment, the device feature subnet includes: a backbone network layer, a segmentation network layer, a classification layer, and a splicing output layer.

[0211] The loss function for the device feature subnet is designed as follows:

[0212]

[0213] in, Represents the classification loss function. This represents the component feature loss function.

[0214] The classification loss function is designed as follows:

[0215] in, This represents the true classification result. This indicates the predicted classification result.

[0216] The component feature loss function is designed as follows:

[0217] in, Represents the sequence of features of the actual components. This represents the predicted component feature sequence.

[0218] In this embodiment, a trained human feature subnet is used to process the human dataset to generate a human feature sequence dataset, and a trained device feature subnet is used to process the device dataset to generate a device feature sequence dataset. The device feature sequence dataset includes a network device feature sequence dataset and a real-world device feature sequence dataset.

[0219] In this embodiment, the loss function of the intent recognition model is designed as follows:

[0220]

[0221] in, Represents classification loss. This indicates the alignment regression loss. This represents the weight of the alignment regression loss. Preferably, the weight of the alignment regression loss is set to 0.3.

[0222] The classification loss function is designed as follows:

[0223]

[0224] in, This represents the true classification result. This indicates the predicted classification result.

[0225] The alignment regression loss function is designed as follows:

[0226]

[0227] in, This represents the true alignment result. This indicates the predicted accuracy result.

[0228] In this embodiment, based on the set of images to be identified in a real scene, the trained human feature subnet and the trained device feature subnet are used to process them respectively to obtain paired data of real human dataset and real device dataset. The intention recognition model is verified based on the paired data, and the intention recognition model is fine-tuned based on the verification results until the verification results meet the training requirements to obtain a trained intention recognition model.

[0229] In some implementations, when the device being used is updated, the device dataset is updated, and the aforementioned intent recognition model training method is executed based on the updated device dataset to update the intent recognition model. This eliminates the need to re-collect and label the combined data of the device and the person, as well as to modify or retrain the person feature subnet. This modular design gives the system efficient iteration and expansion capabilities, enabling it to quickly adapt to the upgrading of retail terminal devices and significantly reducing the long-term maintenance costs of the system.

[0230] The scope of protection for the behavioral intent recognition method and intent recognition model training method in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0231] This application also provides a behavior intention recognition device, which can implement the behavior intention recognition method of this application. However, the implementation device of the behavior intention recognition method of this application includes, but is not limited to, the structure of the behavior intention recognition device listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0232] Please see Figure 9 The image shown is a behavior intent recognition device according to an embodiment of this application.

[0233] like Figure 9 As shown, the behavioral intent recognition device 9 includes:

[0234] The acquisition module 91 is configured to acquire the image to be recognized;

[0235] The human feature extraction module 92 is configured to process the image to be identified based on the human feature subnet to obtain a human feature sequence;

[0236] The device feature extraction module 93 is configured to process the image to be identified based on the device feature subnet to obtain the device feature sequence;

[0237] The feature fusion module 94 is configured to fuse the person feature sequence and the device feature sequence to obtain a fused feature sequence;

[0238] The intent recognition module 95 is configured to process the fused feature sequence using an intent recognition model to obtain intent recognition results.

[0239] It should be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0240] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0241] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0242] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] Figure 10 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 1000 includes at least one processor 101, a memory 102, at least one network interface 103, and a user interface 105. The various components in the device are coupled together via a bus system 104. It is understood that the bus system 104 is used to implement communication between these components. In addition to a data bus, the bus system 104 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 10 The general will label all buses as bus systems.

[0244] The user interface 105 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0245] It is understood that memory 102 can be volatile memory or non-volatile memory, or both. Among them, non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which is used as an external cache.

[0246] In this embodiment, the memory 102 is used to store various types of data to support the operation of the electronic device 100. Examples of such data include any executable program that operates on the electronic device 100, such as the operating system 1021 and application programs 1022; the operating system 1021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks.

[0247] The methods disclosed in the embodiments of this application described above can be applied to processor 101, or implemented by processor 101. Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 101 or by instructions in software form.

[0248] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0249] This application embodiment may also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the flow or function according to the embodiments of this application is generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0250] When the computer program product is executed by a computer, the computer performs the methods described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing methods are required, the computer program product can be downloaded and executed on the computer.

[0251] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0252] In summary, this application provides a method and apparatus for behavioral intent recognition, a training method, a medium, and a device. By decoupling the overall recognition process into independent "person feature subnets" and "device feature subnets" for separate training, it eliminates the reliance on scarce "person-device" joint annotation data found in traditional solutions. The personnel network can be trained using publicly available and abundant human or hand datasets; the device network only requires images of the device itself, which can be obtained at low cost through web scraping. This design frees model construction from the constraints of difficult-to-collect specific combinations of samples, significantly reducing data annotation costs and project startup barriers. When new devices appear that are not present in the training set, only the device feature subnet needs to be updated, and the intent recognition model can achieve generalized recognition without joint samples. By fusing person feature sequences and device feature sequences and outputting visual information such as attention weights, it clearly indicates the key visual evidence upon which the final decision is based, enhancing the credibility of the results and providing clear and direct evidence for manual review, diagnosis of misjudgments, and model optimization. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0253] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for recognizing behavioral intent, characterized in that, The method includes: Acquire the image to be recognized; The image to be identified is processed based on a person feature subnet to obtain a person feature sequence; and The image to be identified is processed based on the device feature subnet to obtain a device feature sequence; The character feature sequence and the device feature sequence are fused to obtain a fused feature sequence; The fused feature sequence is processed using an intent recognition model to obtain intent recognition results.

2. The behavioral intent recognition method according to claim 1, characterized in that, The image to be identified is processed based on a person feature subnet to obtain a person feature sequence, including: Human body regions are detected and cropped from the image to be identified; The human body region image is used to extract features through the backbone network to obtain an initial human body feature map; The initial human feature map is processed by a key point prediction network to obtain several key point heatmaps, and key point feature sequences and orientation feature sequences are obtained based on the several key point heatmaps. The initial human feature map is segmented into hand regions using a segmentation network to obtain hand region images, and a gesture feature sequence is obtained based on the hand region images. The key point feature sequence, the direction feature sequence, and the gesture feature sequence are concatenated to obtain the character feature sequence.

3. The behavioral intent recognition method according to claim 2, characterized in that, Based on several key point heatmaps, key point feature sequences and orientation feature sequences are obtained, including: The soft maximum position and variance of several key point heatmaps are calculated respectively to obtain the sub-pixel coordinate set and confidence score set of the key points; The sub-pixel coordinate set is weighted or filtered based on the confidence score set to obtain an optimized key point data set; and wrist and elbow coordinates are filtered from the sub-pixel coordinate set, and forearm direction vector is calculated based on the wrist and elbow coordinates. The key point data set is mapped using a multilayer perceptron to obtain a key point feature sequence; and the forearm direction vector is mapped using a multilayer perceptron to obtain a direction feature sequence.

4. The behavioral intent recognition method according to claim 1, characterized in that, The image to be identified is processed based on a device feature subnet to obtain a device feature sequence, including: The device area image is detected and cropped from the image to be identified; The device region image is used to extract features through the backbone network to obtain the initial feature map of the device; The initial feature map of the device is segmented and statistically analyzed using a segmentation network to obtain a component probability map; Based on the component probability map, the component feature sequence and main axis direction sequence of each component are obtained respectively; The initial feature map of the device is classified using a classifier to obtain the device category; The component feature sequence, the spindle direction sequence, and the equipment category are spliced ​​together to obtain the equipment feature sequence.

5. The behavioral intent recognition method according to claim 1, characterized in that, The fused feature sequence is obtained by fusing the person feature sequence and the device feature sequence, including: Cross-attention learning is performed on the person feature sequence, where the person feature sequence is used as the query and the device feature sequence is used as the key and value, to obtain the updated person feature sequence. Cross-attention learning is performed on the device feature sequence, where the device feature sequence is used as the query and the person feature sequence is used as the key and value, to obtain the updated device feature sequence. The updated character feature sequence and the updated device feature sequence are concatenated to obtain a concatenated feature sequence; The spliced ​​feature sequence is input into an attention encoder for further encoding to obtain a fused feature sequence.

6. The behavioral intent recognition method according to claim 1, characterized in that, The intent recognition result includes: intent classification result and alignment regression value. The fused feature sequence is processed using an intent recognition model to obtain the intent recognition result, including: The fused feature sequence is pooled to obtain a global feature vector; The global feature vector is input into the classification subnetwork and the regression subnetwork, respectively; The classification subnetwork outputs the intention classification result, and the regression subnetwork outputs the alignment regression value.

7. The behavioral intent recognition method according to claim 6, characterized in that, The process of processing the fused feature sequence using an intent recognition model to obtain intent recognition results further includes: Obtain the geometric feature sequence; The global feature vector is fused with the geometric feature sequence to obtain the regression input features; The regression input features are input into the regression sub-network to obtain the angle between the device and the person's hand.

8. A method for training an intent recognition model, characterized in that, The method includes: Separately acquire a person dataset and a device dataset; wherein, the device dataset is a hybrid dataset, including device images obtained from the network and real-world images; A person feature subnet is trained based on the person dataset, and a device feature subnet is trained based on the device dataset. Using the trained person feature subnet and the trained device feature subnet, the person dataset and the device dataset are processed offline to generate person feature sequence dataset and device feature sequence dataset, respectively. The intent recognition model is trained based on the person feature sequence dataset and the device feature sequence dataset; A set of images to be identified in a real scene is obtained, and the set of images to be identified is processed using the person feature subnet and the device feature subnet respectively to obtain paired data of real person dataset and real device dataset; The intent recognition model is fine-tuned based on the pairing data; wherein at least some parameters of the person feature subnet and the device feature subnet are frozen.

9. A behavioral intent recognition device, characterized in that, The device includes: The acquisition module is configured to acquire the image to be recognized. The human feature extraction module is configured to process the image to be identified based on a human feature subnet to obtain a human feature sequence; The device feature extraction module is configured to process the image to be identified based on the device feature subnet to obtain a device feature sequence; The feature fusion module is configured to fuse the person feature sequence and the device feature sequence to obtain a fused feature sequence; The intent recognition module is configured to process the fused feature sequence using an intent recognition model to obtain intent recognition results.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the behavioral intent recognition method according to any one of claims 1 to 7 and / or the intent recognition model training method according to claim 8.

11. An electronic device, characterized in that, Including memory and processor; The memory is configured to store computer programs; The processor is configured to execute a computer program stored in the memory, so that the electronic device, when executed, implements the behavioral intent recognition method of any one of claims 1 to 7 and / or the intent recognition model training method of claim 8.

Images