Methods and devices for classifying people

Abstract

This application discloses a method and apparatus for classifying people. The method includes: acquiring voice data and video data of a person to be detected; based on the voice data and video data, using a fusion sub-model to output a first multimodal fusion code of the person to be detected in three dimensions: voice, behavior, and movement trajectory; using a feature extraction sub-model to extract features from the first multimodal fusion code to obtain features of the person to be detected; determining the distance between the features of the person to be detected and each person category feature in the person category feature set; wherein, the person category feature is the feature obtained by clustering the classification vector output by the classification sub-model after training a crowd classification model using a training sample set; determining the target person category feature in the person category feature set based on the distance; and determining the person category corresponding to the target person category feature as the category of the person to be detected.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer vision technology, and in particular to a method and apparatus for classifying people. Background Technology

[0002] In related technologies, the classification of people typically involves two methods: The first method involves acquiring facial information and then classifying people through facial recognition. However, firstly, facial information is difficult to obtain; secondly, even if it can be obtained, the accuracy of facial information captured by a camera is sometimes affected by parameters such as camera distance and angle, thus impacting the accuracy of facial classification. The second method, which relies solely on image data to distinguish people, is limited by significant variations in human movement and clothing, resulting in poor classification results. Therefore, a new method for people classification is needed. Summary of the Invention

[0003] In view of this, the embodiments of this application provide at least one method and apparatus for classifying people.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a person classification method, the method comprising: acquiring voice data and video data of a person to be detected; based on the voice data and the video data, using a fusion sub-model in a crowd classification model to output a first multimodal fusion code of the person to be detected in three dimensions: voice, behavior, and movement trajectory; using a feature extraction sub-model in the crowd classification model to extract features from the first multimodal fusion code to obtain features of the person to be detected; determining the distance between the features of the person to be detected and each person category feature in a person category feature set; wherein, the person category feature is a feature obtained by clustering the classification vector output by the classification sub-model in the crowd classification model after training the crowd classification model with a training sample set, and the classification vector is a vector used to output a classification score; based on the distance, determining a target person category feature in the person category feature set; and determining the person category corresponding to the target person category feature as the category of the person to be detected.

[0006] Secondly, embodiments of this application provide a person classification device, the device comprising: a first acquisition module, configured to acquire voice data and video data of a person to be detected; a first output module, configured to output a first multimodal fusion code of the person to be detected in three dimensions—voice, behavior, and movement trajectory—based on the voice data and the video data, using a fusion sub-model in a crowd classification model; a first extraction module, configured to extract features from the first multimodal fusion code using a feature extraction sub-model in the crowd classification model to obtain features of the person to be detected; a first determination module, configured to determine the distance between the features of the person to be detected and each person category feature in a person category feature set; wherein, the person category feature is a feature obtained by clustering the classification vector output by the classification sub-model in the crowd classification model after training the crowd classification model using a training sample set, and the classification vector is a vector used to output a classification score; a second determination module, configured to determine a target person category feature in the person category feature set based on the distance; and a third determination module, configured to determine the person category corresponding to the target person category feature as the category of the person to be detected.

[0007] In this embodiment, the category of the person to be detected is obtained by determining the distance between the features of the person to be detected and each category feature in the category feature set. Since each category feature in the category feature set is a feature obtained by clustering the classification vector output by a trained crowd classification model, and since the classification vector incorporates information from three dimensions—voice, behavior, and movement trajectory—the clustering process yields more and finer-grained category features compared to clustering using only images or videos as input. This results in a more granular category for the person to be detected.

[0008] Furthermore, the embodiments of this application obtain the category of the person to be detected through voice data and video data, without using facial information. Therefore, it solves the problems mentioned in related technologies, such as the inconvenience of obtaining facial information, the inaccuracy of facial information affecting the accuracy of person classification through facial information, and the poor classification results caused by simply using image data to distinguish groups of people due to the large variations in human movements, clothing and other features.

[0009] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0011] Figure 1 A schematic diagram illustrating the implementation process of a person classification method provided in this application embodiment;

[0012] Figure 2A A schematic diagram illustrating the implementation process of a training method for a population classification model provided in this application embodiment;

[0013] Figure 2B A schematic diagram illustrating the implementation process of step S201 provided in an embodiment of this application;

[0014] Figure 3A A schematic diagram illustrating the implementation flow of a method for determining a first speech code provided in an embodiment of this application;

[0015] Figure 3B A schematic diagram illustrating the implementation flow of a method for determining the encoding of the first line, provided in an embodiment of this application;

[0016] Figure 3C A schematic diagram illustrating the implementation flow of a method for determining a first movement trajectory code, provided in an embodiment of this application;

[0017] Figure 4 This is a schematic diagram illustrating the implementation process of annotating a set of speech samples and a set of video samples, provided in an embodiment of this application.

[0018] Figure 5A A schematic diagram illustrating the implementation process of another training method for a population classification model provided in this application embodiment;

[0019] Figure 5B A schematic diagram illustrating a common behavior of a shop assistant, provided as an embodiment of this application;

[0020] Figure 5C A schematic diagram of a staff heat zone provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of the composition structure of a person classification device provided in an embodiment of this application;

[0022] Figure 7 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0025] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0027] This application provides a method for classifying people, which can be executed by a processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 This is a schematic diagram illustrating the implementation process of a character classification method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps S101 to S106:

[0028] Step S101: Obtain the voice and video data of the person to be detected;

[0029] Here, voice data and video data refer to voice data and video data collected from the person to be detected at the same time and in the same space. That is, voice data corresponds to video data. For example, voice data and video data can be data obtained by simultaneously collecting voice and image data from a person to be detected at an Automobile Sales Service Shop (4S). Among them, voice data can be obtained by recording with a microphone, and video data can be obtained by capturing images with a camera.

[0030] Step S102: Based on the voice data and the video data, the first multimodal fusion code of the person to be detected in the three dimensions of voice, behavior and movement trajectory is output using the fusion sub-model in the crowd classification model;

[0031] Here, the first multimodal fusion coding refers to the coding obtained by fusing the coding of three dimensions: speech, behavior, and movement trajectory. This coding is used as input to the feature extraction sub-model in the crowd classification model for feature extraction of individuals. In implementation, speech data can first be converted into speech-dimensional coding, and video data into behavior-dimensional and movement trajectory-dimensional coding; then, the speech-dimensional coding, behavior-dimensional coding, and movement trajectory-dimensional coding are fused to obtain the first multimodal fusion coding.

[0032] The speech dimension encoding is a code that the algorithm can recognize, obtained by converting speech data. The behavior dimension encoding is a code that the algorithm can recognize, obtained from the detection results of behavior detection on images in video data. For example, the behavior dimension encoding can be the vector output by a classification network after behavior detection on images in video data, used to output the classification score. For example, in the case of a Visual GeometryGroup (VGG) classification network, if the sub-network before the output classification score is a fully connected layer, and the fully connected layer is used to output the classification score, then the behavior dimension encoding is the vector output by the fully connected layer.

[0033] The motion trajectory dimension encoding is the algorithm-recognizable encoding used to represent the motion trajectory in the detection result obtained after object tracking and detection of images in video data. For example, the motion trajectory dimension encoding can be obtained by concatenating the vector output by the object detection box after object tracking and detection of images in video data through an object tracking network with the vector transformed from the center point coordinates of the object detection box. For example, in the case of a multi-domain network (MDNet) where the sub-network before the output object detection box is a fully connected layer, and the fully connected layer is used to output the object detection box, then the motion trajectory dimension encoding can be obtained by concatenating the vector output by the fully connected layer with the vector transformed from the center point coordinates of the object detection box.

[0034] Since the encoding of the speech dimension, behavior dimension, and motion trajectory dimension are encoded in three different spaces, they cannot be directly fused. Therefore, the fusion sub-model can utilize, for example, a multilayer perceptron to transform the encodings of the speech dimension, behavior dimension, and motion trajectory dimension into the same implicit space. Then, the encodings of the three dimensions are fused in the same space to facilitate subsequent feature extraction from the fused encoding, thereby obtaining features that fuse speech, behavior, and motion trajectory information. This results in features containing more dimensional information, enabling the differentiation of finer-grained categories, and achieving higher category accuracy. This application does not limit the method for obtaining the first multimodal fusion encoding.

[0035] Step S103: Use the feature extraction sub-model in the crowd classification model to extract features from the first multimodal fusion encoding to obtain the features of the person to be detected;

[0036] Here, the feature extraction sub-model is used to extract features from the first multimodal fusion encoding to obtain the features of the person to be detected. In some embodiments, the feature extraction sub-model can be a model containing a convolutional network. This application does not limit the structure of the feature extraction sub-model; it can be configured with reference to related technologies.

[0037] Step S104: Determine the distance between the features of the person to be detected and each person category feature in the person category feature set; wherein, the person category feature is the feature obtained by clustering the classification vector output by the classification sub-model in the crowd classification model after training the crowd classification model with the training sample set, and the classification vector is the vector used to output the classification score;

[0038] Here, the distance between the features of individuals in the population to be detected and the features of each individual category in the set of individual category features can be the cosine distance.

[0039] The classification sub-model in the crowd classification model is used to classify the human features extracted by the feature extraction sub-model, thereby obtaining a classification vector. Since the human features extracted by the feature extraction sub-model include information in three dimensions: voice, behavior, and movement trajectory, the classification vector obtained by the classification sub-model in the crowd classification model also contains more dimensions of information. After clustering, more categories and more fine-grained human category features can be obtained. These human category features are the features corresponding to the cluster centers obtained after clustering.

[0040] In implementation, a training sample set can be used to train the population classification model to obtain classification vectors for multiple individuals, thereby obtaining more categories of individual features. In some embodiments, the classification sub-model may include network models such as VGG, Googlenet, Residual Network 50 (ResNet50), and MobileNetV2. This application does not limit the structure of the classification sub-model; it can be configured with reference to related technologies.

[0041] The classification vector is the vector used to output the classification score. For example, in the case of a VGG network as the classification sub-model, if the layer preceding the output classification score is a fully connected layer, and the fully connected layer is used to output the classification score, then the classification vector can be the vector output by the fully connected layer.

[0042] Step S105: Based on the distance, determine the target person category features in the set of person category features;

[0043] The following example illustrates the category features of people, including: store clerk features, first-level interested customer features, second-level interested customer features, and third-level interested customer features. For instance, if the store clerk features have the smallest distance from the features of people in the population to be tested, then the target person category feature is the store clerk features.

[0044] Step S106: Determine the category of the person corresponding to the target person category feature as the category of the person to be detected.

[0045] The following explanation continues using the example of store clerk as the target person category feature. If the person category corresponding to the store clerk feature is store clerk, then the category of people in the population to be detected is store clerk.

[0046] In this embodiment, the category of the person to be detected is obtained by determining the distance between the features of the person to be detected and each category feature in the category feature set. Since each category feature in the category feature set is a feature obtained by clustering the classification vector output by a trained crowd classification model, and since the classification vector incorporates information from three dimensions—voice, behavior, and movement trajectory—the clustering process yields more and finer-grained category features compared to clustering using only images or videos as input. This results in a more granular category for the person to be detected.

[0047] Furthermore, the embodiments of this application obtain the category of the person to be detected through voice data and video data, without using facial information. Therefore, it solves the problems mentioned in related technologies, such as the inconvenience of obtaining facial information, the inaccuracy of facial information affecting the accuracy of person classification through facial information, and the poor classification results caused by simply using image data to distinguish groups of people due to the large variations in human movements, clothing and other features.

[0048] In some embodiments, the training sample set includes a speech sample set and a video sample set, such as... Figure 2A As shown, the population classification model is trained using the following method, including steps S201 to S205:

[0049] Step S201: Based on the acquired set of voice samples and the set of video samples with labeled crowd categories, the second multimodal fusion code of the crowd in the three dimensions of voice, behavior and movement trajectory is output using the fusion sub-model;

[0050] Here, the speech sample set and video sample set can be sets of speech samples and video samples collected from crowds at the same time and in the same space. That is, the speech sample set corresponds to the video sample set. The labeled categories can be any categories, such as shop assistants and customers, bosses and employees, students and teachers, etc.

[0051] Step S202: Use the feature extraction sub-model to extract features from the second multimodal fusion encoding to obtain the features of the people in the crowd;

[0052] Step S203: Use the classification sub-model to classify the features of the people in the crowd to obtain the classification result;

[0053] Step S204: Based on the classification results and the labeled population category tags, determine the loss function of the population classification model;

[0054] Here, the loss function of the crowd classification model can be the cross-entropy loss function. Step S204 can be implemented by using the obtained classification results and the labeled crowd category labels, substituting them into the cross-entropy loss function to obtain the loss function of the crowd classification model.

[0055] Step S205: Based on the loss function, train the population classification model until convergence.

[0056] Here, step S205 can be implemented by adjusting the network parameters of the crowd classification model until convergence using the obtained loss function, thereby obtaining a well-trained crowd classification model.

[0057] In this embodiment, firstly, a fusion sub-model is used to output a second multimodal fusion code of the crowd in three dimensions: speech, behavior, and movement trajectory. Then, a feature extraction sub-model is used to extract features from the second multimodal fusion code to obtain the features of individuals in the crowd. Next, a classification sub-model is used to classify the features of individuals in the crowd to obtain the classification result. Then, based on the classification result, the loss function of the crowd classification model is determined. Finally, based on the loss function, the crowd classification model is trained until convergence to obtain the trained model.

[0058] As can be seen, because the second multimodal fusion coding integrates information from three dimensions of speech, behavior, and movement trajectory, the features obtained after feature extraction also contain information from these three dimensions. This results in features containing more dimensional information, enabling finer-grained classification through the sub-model and achieving higher accuracy.

[0059] In some embodiments, the fusion sub-model includes an encoding module and a fusion module, correspondingly, such as Figure 2B As shown, the implementation of step S201, "based on the acquired set of voice samples and the set of video samples with labeled population categories, outputting the second multimodal fusion code of the population in the three dimensions of voice, behavior, and movement trajectory using the fusion sub-model," may include the following steps S2011 to S2013:

[0060] Step S2011: Based on the acquired set of voice samples and the set of video samples with labeled crowd categories, the encoding module outputs the first voice code of the crowd in the voice dimension, the first behavior code in the behavior dimension, and the first movement trajectory code in the movement trajectory dimension.

[0061] Here, the first speech code refers to the code that the algorithm can recognize in the speech sample set, i.e., the speech dimension code mentioned above. The first action code refers to the code that the algorithm can recognize in the detection result obtained after action detection on the images in the video sample set, i.e., the action dimension code mentioned above. The first motion trajectory code refers to the code that the algorithm can recognize in the detection result obtained after target tracking and detection on the images in the video sample set, i.e., the motion trajectory dimension code mentioned above.

[0062] In some embodiments, the dimensions of the first speech encoding, the first behavior encoding, and the first motion trajectory encoding may each include two dimensions (width dimension and height dimension). For example, the dimension of the first speech encoding may be 1*n, the dimension of the first behavior encoding may be 1*n, and the dimension of the first motion trajectory encoding may be k*(n+m). In other embodiments, the dimensions of the first speech encoding, the first behavior encoding, and the first motion trajectory encoding may each include three dimensions (channel dimension, width dimension, and height dimension). For example, the dimension of the first speech encoding may be g*1*n, the dimension of the first behavior encoding may be g*1*n, and the dimension of the first motion trajectory encoding may be g*k*(n+m). The embodiments of this application do not limit the number of dimensions for the first speech encoding, the first behavior encoding, and the first motion trajectory encoding.

[0063] Step S2012: Using the fusion module, the first speech code, the first behavior code, and the first movement trajectory code are respectively converted into the same implicit space to obtain the second speech code, the second behavior code, and the second movement trajectory code;

[0064] Here, step S2012 can be implemented by using a multilayer perceptron to transform the first speech code, the first behavior code, and the first action trajectory code into the same implicit space, so that the codes of the three dimensions can be spliced ​​together to obtain the second multimodal fusion code.

[0065] In some embodiments, the second speech code, the second behavior code, and the second movement trajectory code may each include two dimensions, and at least one dimension may have the same dimensionality. In other embodiments, the second speech code, the second behavior code, and the second movement trajectory code may each include three dimensions, and at least two dimensions may have the same dimensionality. This allows the second speech code, the second behavior code, and the second movement trajectory code to be concatenated along dimensional directions with different dimensions to obtain the second multimodal fusion code.

[0066] Step S2013: Concatenate the second speech code, the second behavior code, and the second movement trajectory code to obtain the second multimodal fusion code.

[0067] Here, when the second speech code, the second behavior code, and the second motion trajectory code all include two dimensions, the fusion process can be performed by concatenating along dimensional directions with different dimensions to obtain the second multimodal fusion code. The following explanation uses an example where the second speech code has a dimension of 1*n, the second behavior code has a dimension of 1*n, and the second motion trajectory code has a dimension of k*n to illustrate the method for forming the second multimodal fusion code. Where the second speech code, the second behavior code, and the second motion trajectory code have different height dimensions, concatenation is performed along the height dimension direction of the second speech code, the second behavior code, and the second motion trajectory code to obtain a second multimodal fusion code with a dimension of (2+k)*n. The fusion method when the second speech code, the second behavior code, and the second motion trajectory code all include three dimensions will not be elaborated here.

[0068] In this embodiment, the encoding module first outputs the first speech code in the speech dimension, the first behavior code in the behavior dimension, and the first movement trajectory code in the movement trajectory dimension. Then, the fusion module converts the first speech code, the first behavior code, and the first movement trajectory code into the same implicit space to obtain the second speech code, the second behavior code, and the second movement trajectory code. Finally, the second speech code, the second behavior code, and the second movement trajectory code are concatenated to obtain the second multimodal fusion code, thereby achieving the fusion of multimodal codes.

[0069] In some embodiments, the encoding module includes: a speech feature extraction subnetwork, an action detection subnetwork, and a target tracking and detection subnetwork. Correspondingly, the implementation of step S2011 may include the following steps S21a to S21c:

[0070] Step S21a: Based on the speech sample set of the population, the speech feature extraction sub-network is used to output the first speech code of the population in the speech dimension;

[0071] Here, the speech feature extraction subnetwork is used to extract speech features from speech samples in the speech sample set. In some embodiments, the speech feature extraction subnetwork can be a Wav 2Vec network. The first speech encoding can be the vector output by the previous subnetwork before the speech text output by the speech feature extraction subnetwork.

[0072] Correspondingly, such as Figure 3A As shown, the implementation of step S21a may include the following steps S301a and S302a:

[0073] Step S301a: Use the speech feature extraction sub-network to extract speech features from the speech sample set to obtain a first vector, wherein the first vector is a vector used to output speech text;

[0074] Here, the dimension of the first vector can be 1*n. For example, if the speech feature extraction subnetwork is a Wav 2Vec network, and the previous subnetwork in the Wav 2Vec network that outputs speech text is used to output speech text, then the first vector is the vector output by the aforementioned previous subnetwork.

[0075] Step S302a: Determine the first vector as the first speech code of the population in the speech dimension.

[0076] It should be noted that the first speech code can be obtained through a pre-trained speech feature extraction sub-network (e.g., Wav 2Vec) in related technologies. This application embodiment does not limit the method of obtaining the first speech code.

[0077] Step S21b: Based on the video sample set of the crowd, the behavior detection sub-network is used to output the first behavior code of the crowd in the behavior dimension;

[0078] Here, the behavior detection subnetwork is used to detect behavior in images within the video sample set. In some embodiments, the behavior detection subnetwork may include a classification network, such as VGG, Googlenet, ResNet50, MobileNetV2, etc. In implementation, the behavior detection subnetwork can be trained using common behaviors of different crowd categories, enabling it to distinguish whether the behavior of people in the video sample set is a common behavior of a certain category, thus helping to identify the crowd category. For example, common behaviors of shop assistants include: raising their hand to greet customers; extending their arms downwards to introduce products; and holding sales tools with both hands while communicating with customers. Therefore, when training the behavior detection subnetwork, it can be trained using these three behaviors: raising their hand to greet customers; extending their arms downwards to introduce products; and holding sales tools with both hands while communicating with customers, so that the behavior detection subnetwork can distinguish whether the behavior of people in the video sample set belongs to a shop assistant.

[0079] In some embodiments, such as Figure 3B As shown, the implementation of step S21b may include the following steps S301b and S302b:

[0080] Step S301b: Use the behavior detection sub-network to perform behavior dimension classification detection on the images in the video sample set to obtain a second vector, wherein the second vector is a vector used to output the classification score;

[0081] Here, with the classification network being a VGG network, the sub-network before the one that outputs the classification score is a fully connected layer. Since the fully connected layer is used to output the classification score, the second vector can be the vector output by the fully connected layer.

[0082] In some embodiments, step S301b can be implemented by stitching together the images in the video sample set and outputting a second vector to obtain the second multimodal fusion code. The dimension of the second vector can be 1*n.

[0083] Step S302b: Determine the second vector as the first behavior code of the population in the behavior dimension.

[0084] Step S21c: Based on the video sample set of the crowd, the target tracking and detection sub-network is used to output the first action trajectory code of the crowd in the action trajectory dimension.

[0085] Here, the target tracking and detection subnetwork is used to track targets in images within the video sample set. In some embodiments, the target tracking and detection subnetwork may include an MDNet network, a Siamese network, etc. In implementation, a pre-trained target tracking and detection subnetwork from related technologies can be used to obtain the first motion trajectory encoding.

[0086] In some embodiments, such as Figure 3C As shown, the implementation of step S21c may include the following steps S301c to S303c:

[0087] Step S301c: Use the target tracking and detection sub-network to perform target tracking and detection on the images in the video sample set in the dimension of action trajectory to obtain a third vector and a target detection box, wherein the third vector is a vector used to output the target detection box;

[0088] Here, when the target tracking and detection subnetwork is an MDNet network, the layer preceding the output target detection box is a fully connected layer. Since the fully connected layer is used to output the target detection box, the third vector can be the vector output by the fully connected layer. In some embodiments, the dimension of the third vector can be 1*n.

[0089] Step S302c: Obtain the coordinates of the center point of the target detection box, and convert the center point coordinates into a fourth vector;

[0090] Here, the center point coordinates of the target detection box can be (x, y). In some embodiments, the method for converting the center point coordinates into a fourth vector may include: converting the center point coordinates into a binary value with m bits, that is, corresponding to a fourth vector with a dimension of 1*m, thereby obtaining the fourth vector. The first half of the numbers (i.e., the first m / 2 numbers) represents the x-coordinate, and the second half of the numbers (i.e., the last m / 2 numbers) represents the y-coordinate.

[0091] Step S303c: The concatenated third vector and the fourth vector are determined as the first movement trajectory code of the crowd in the movement trajectory dimension.

[0092] Here, step S303c can be implemented by first concatenating the third and fourth vectors to obtain the first action trajectory code. The method for concatenating the third and fourth vectors can include: if both the third and fourth vectors have two dimensions, they can be transformed into vectors with the same dimension in at least one dimension using a dimension transformation method; then, they are concatenated along the dimensional directions with different dimensions to obtain the concatenated third and fourth vectors. The following example illustrates the method for concatenating the third and fourth vectors using a third vector with a dimension of 1*n and a fourth vector with a dimension of 1*m. If the third and fourth vectors have different dimensions in the height dimension, they are concatenated along the height dimension direction to obtain concatenated third and fourth vectors with a dimension of 1*(n+m). If both the third and fourth vectors have three dimensions, they can be transformed into vectors with the same dimension in at least two dimensions using a dimension transformation method; then, they are concatenated along the dimensional directions with different dimensions to obtain the concatenated third and fourth vectors. Further examples are not provided here.

[0093] In some embodiments, such as Figure 4 As shown, the method further includes the following steps S401 to S403:

[0094] Step S401: Obtain the classification vector output from the classification sub-model;

[0095] Here, as above, when the classification sub-model is a VGG network model, if the sub-network before the output classification score is a fully connected layer, then the classification vector can be the vector output by the fully connected layer.

[0096] Step S402: Perform clustering processing on the classification vector to obtain the updated categories of people in the crowd;

[0097] Here, the clustering algorithms can include: K-means clustering, mean-shift clustering, clustering algorithms that use Gaussian mixture models for maximum expectation estimation, agglomerative hierarchical clustering, etc., without limitation.

[0098] In some embodiments, the method provided in this application is used to distinguish between shop assistants and customers. After clustering, the updated categories obtained may include shop assistants, first-level interested customers, second-level interested customers, and third-level interested customers. Correspondingly, the implementation of step S402 may include the following steps S4021 and S4022:

[0099] Step S4021: Perform clustering processing on the classification vector to obtain at least one person category feature, wherein the person category feature is the feature corresponding to the cluster center obtained after clustering processing;

[0100] Here, since the classification vector incorporates information from three dimensions—voice, behavior, and movement trajectory—the clustering process in step S4021 yields at least one person category feature. Compared to clustering using only images or videos as input, this method provides more and finer-grained person category features.

[0101] Step S4022: Based on the at least one person category feature, determine the updated category of the people in the crowd, including: shop assistant, first-level interested customer, second-level interested customer and third-level interested customer.

[0102] Here, step S4022 can be implemented through manual tagging to determine the updated category of the customer from at least one customer category feature, including: shop assistant, first-level interested customer, second-level interested customer, and third-level interested customer. The first-level interested customer, second-level interested customer, and third-level interested customer are different categories of customers obtained by classifying their level of interest. For example, a first-level interested customer is a very interested customer, a second-level interested customer is a moderately interested customer, and a third-level interested customer is a not interested customer.

[0103] Step S403: Based on the updated categories, re-label the speech sample set and the video sample set.

[0104] Here, the implementation of step S403 can be understood as follows: after obtaining more granular categories through clustering, the voice sample set and video sample set are re-labeled, and then the re-labeled sample set is used to train the crowd classification model. Then, more granular categories are obtained through clustering, and the voice sample set and video sample set are re-labeled. This process is repeated to obtain the required human category features.

[0105] In this embodiment, the updated categories of people in the crowd are obtained by clustering the classification vectors output from the classification sub-model. Since the classification vectors incorporate information from three dimensions—voice, behavior, and movement trajectory—the clustering process yields more and finer-grained category features compared to clustering using only images or videos as input. Re-labeling the voice and video sample sets with the updated categories to train the crowd classification model allows it to output even more categories of people.

[0106] The following describes the application of the training method of the crowd classification model provided in this application embodiment in a real-world scenario, taking the classified crowd as store employees and customers as an example.

[0107] Modern life is filled with all sorts of stores, such as clothing stores, restaurants, mobile phone experience stores, and car showrooms. With technological advancements, store management is becoming increasingly intelligent and digitalized, making personalized customer analysis a crucial approach. From a technical perspective, personalized customer analysis first requires identifying customer categories, then analyzing the behavioral characteristics of different individuals, and calculating numerical results that aid in store management. Traditionally, this has been achieved by recording employees' facial information for relatively accurate customer and employee differentiation. However, facial information is difficult to obtain, and variations in camera angles and other parameters across different stores lead to inaccurate facial data. Therefore, using facial information to determine customer categories is an unusable solution.

[0108] This application proposes a training method for a population classification model. This method can be applied to any offline store, eliminating the need for the cumbersome process of pre-entering biometric information.

[0109] The training method for the crowd classification model proposed in this application combines three different modalities of data: voice, image (i.e., the aforementioned behavioral dimension), and time (i.e., the aforementioned movement trajectory dimension). This method can be directly applied to different types of stores.

[0110] The method provided in this application includes:

[0111] 1) Collect the voice (i.e., the above voice sample set) and image data (i.e., the above video sample set) of each person in the store. The voice data is collected through a microphone, and the image data is collected through a surveillance camera.

[0112] 2) Collect data on each person's temporal modality (i.e., the video sample set mentioned above). This data can be used to estimate the two features of each person's movement trajectory and residence hot zone over a period of time (i.e., the encoding of the movement trajectory dimension mentioned above) through algorithms.

[0113] 3) Use three MLPs (Multilayer Perceptrons) (i.e. the fusion module mentioned above) to transform the three different types of data into the implicit space, and then concatenate them to obtain the fusion code (i.e. the second multimodal fusion code mentioned above);

[0114] 4) Regress the fused code using a neural network (i.e., the feature extraction sub-model and classification sub-model mentioned above) to obtain the feature code (i.e., the classification vector mentioned above) for each person;

[0115] 5) Cluster the feature codes to obtain the clustering results, which include the features of each category and the category name (i.e., the updated category as described above).

[0116] By following the above five steps, automatic population classification can be achieved. The number of cluster centers in the final step is the number of population categories.

[0117] In practical applications, the number of customer categories within a month can be calculated based on historical data (e.g., voice and image data from within the store over a month), and the clustering results can be saved. Then, the customer categories corresponding to these categories are manually defined, such as store clerks, highly interested customers, moderately interested customers, and uninterested customers (i.e., the updated categories mentioned above). At any given time, the feature codes of store personnel (i.e., the features of the person to be detected) are calculated in real-time. The distance between the person's feature code and the existing clustering results from the previous month (i.e., the features of each person category in the aforementioned set of person category features) is then calculated to identify the current person's category. The backend makes a service decision based on the person's category. If the person is a store clerk, no action is taken; if the person is a highly interested customer, a capable and communicative store clerk is recommended to assist them.

[0118] This application also provides a method for training a population classification model, such as... Figure 5A As shown, the method includes the following steps S501 to S504:

[0119] Step S501: Acquire speech data (i.e., the above-mentioned speech sample set), image data (i.e., the above-mentioned video sample set), and time data (i.e., the above-mentioned video sample set) respectively;

[0120] Step S502: For each person in the store, three different modalities of data are obtained, and each data is processed by an MLP (i.e., the above-mentioned fusion module) to obtain its own encoding. These three encodings (i.e., the above-mentioned second speech encoding, second behavior encoding and second movement trajectory encoding) are concatenated to form a multimodal encoding (i.e., the above-mentioned second multimodal fusion encoding).

[0121] Here, the process of forming speech modality coding (i.e., the second speech coding mentioned above) is as follows: First, audio data (i.e., the speech sample set mentioned above) is acquired through a microphone; then, speech coding is performed to obtain the first speech coding mentioned above; finally, MLP is used to obtain speech modality coding, which is a 1*n dimension vector.

[0122] The formation process of image modality coding (i.e., the second behavior coding mentioned above) is as follows: First, images (i.e., images from the video sample set mentioned above) are acquired in real time through a camera; then, behavior detection is performed; finally, the behavior detection result (i.e., the vector used by the behavior detection sub-network to output the classification score) is converted into image modality coding through an MLP. This coding is also a 1*n dimensional vector. Behavior detection refers to the detection of specific actions. For example, common actions of a salesperson include the action of raising their hand to greet passersby at the entrance / exit, the action of a salesperson introducing car information to a customer, and the action of a salesperson holding sales tools while communicating with a customer. Figure 5B The figures show common actions of shop assistants. Figure (a) shows a shop assistant greeting passersby by raising their hand at the entrance / exit; Figure (b) shows a shop assistant introducing car information to a customer; and Figure (c) shows a shop assistant holding sales tools and communicating with a customer.

[0123] The process of temporal modal coding (i.e., the second motion trajectory coding mentioned above) is as follows: First, images over a period of time are acquired, i.e., video data (the aforementioned video sample set); then, target tracking is performed; finally, the target tracking result (i.e., the vector used by the target tracking and detection sub-network to output the target detection box) is converted into temporal modal coding using an MLP. This coding is a k*(n+m) dimensional vector. Target tracking refers to the real-time detection of each person's position. Since it is video, over a period of time, each person's activity area within the store will be formed, i.e., a hotspot. Figure 5C As shown, the thick black solid line 501 represents the store entrance and exit, and the thick gray line 502 represents the hot zone. It can be seen that this hot zone likely belongs to the store employees, as it is primarily located along the connecting passageways between the entrance / exit, meeting area, and office.

[0124] Step S503: Regress the multimodal encoding using a neural network (i.e., the feature extraction sub-model and classification sub-model mentioned above) to generate feature encoding (i.e., the classification vector mentioned above).

[0125] At this point, each person has a corresponding feature code.

[0126] Step S504: Cluster the feature codes of all people in the store. The number of cluster centers is the number of population categories.

[0127] When classifying people, an individual's feature code (i.e., the features of the person to be detected mentioned above) is closest to which class center feature (i.e., each person category feature in the set of person category features mentioned above) belongs to the corresponding person category.

[0128] In this embodiment, firstly, by combining data from three different modalities, automatic customer classification is achieved for offline stores, resulting in higher discriminative power and finer-grained differentiation. For example, the differentiated categories include store staff and customers, or even fine-grained customer categories (such as degree of interest), thereby helping to improve the service efficiency and quality of stores. Secondly, since the data from the three modalities cannot be concatenated, this embodiment uses MLP to transform the data from the three modalities into an implicit space before concatenation, making the feature encoding of each person more comprehensive, thereby improving the discrimination effect.

[0129] Based on the foregoing embodiments, this application provides a person classification device, which includes various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0130] Figure 6 This is a schematic diagram of the composition structure of a person classification device provided in an embodiment of this application, as shown below. Figure 6 As shown, the character classification device 600 includes: a first acquisition module 610, a first output module 620, a first extraction module 630, a first determination module 640, a second determination module 650, and a third determination module 660, wherein:

[0131] The first acquisition module 610 is used to acquire the voice data and video data of the person to be detected;

[0132] The first output module 620 is used to output the first multimodal fusion code of the person to be detected in three dimensions: voice, behavior and movement trajectory, based on the voice data and the video data and using the fusion sub-model in the crowd classification model.

[0133] The first extraction module 630 is used to extract features from the first multimodal fusion encoding using the feature extraction sub-model in the crowd classification model to obtain the features of the person to be detected.

[0134] The first determining module 640 is used to determine the distance between the features of the person to be detected and each person category feature in the person category feature set; wherein, the person category feature is the feature obtained by clustering the classification vector output by the classification sub-model in the crowd classification model after training the crowd classification model with the training sample set, and the classification vector is the vector used to output the classification score;

[0135] The second determining module 650 is used to determine the target person category feature in the person category feature set based on the distance;

[0136] The third determining module 660 is used to determine the category of the person corresponding to the target person category feature as the category of the person to be detected.

[0137] In some embodiments, the apparatus further includes: a second output module, configured to output a second multimodal fusion code of the crowd in three dimensions—voice, behavior, and movement trajectory—based on the acquired set of voice samples and the set of video samples labeled with crowd category tags, using the fusion sub-model; a second extraction module, configured to extract features from the second multimodal fusion code using the feature extraction sub-model to obtain features of individuals in the crowd; a classification module, configured to classify the features of individuals in the crowd using the classification sub-model to obtain a classification result; a fourth determination module, configured to determine the loss function of the crowd classification model based on the classification result and the labeled crowd category tags; and a training module, configured to train the crowd classification model until convergence based on the loss function.

[0138] In some embodiments, the fusion sub-model includes an encoding module and a fusion module. Correspondingly, the second output module includes: an output sub-module, used to output, based on the acquired set of voice samples labeled with audience category tags and the set of video samples, the first voice code in the voice dimension, the first behavior code in the behavior dimension, and the first movement trajectory code in the movement trajectory dimension of the audience using the encoding module; a transformation sub-module, used to use the fusion module to transform the first voice code, the first behavior code, and the first movement trajectory code to the same implicit space respectively, to obtain a second voice code, a second behavior code, and a second movement trajectory code; and a splicing sub-module, used to splice the second voice code, the second behavior code, and the second movement trajectory code to obtain the second multimodal fusion code.

[0139] In some embodiments, the encoding module includes: a speech feature extraction subnetwork, a behavior detection subnetwork, and a target tracking and detection subnetwork. Correspondingly, the output submodule includes: a first output unit, used to output a first speech code of the crowd in the speech dimension based on a set of speech samples of the crowd using the speech feature extraction subnetwork; a second output unit, used to output a first behavior code of the crowd in the behavior dimension based on a set of video samples of the crowd using the behavior detection subnetwork; and a third output unit, used to output a first movement trajectory code of the crowd in the movement trajectory dimension based on a set of video samples of the crowd using the target tracking and detection subnetwork.

[0140] In some embodiments, the first output unit includes: a speech feature extraction subunit, configured to extract speech features from the speech sample set using the speech feature extraction subnetwork to obtain a first vector, wherein the first vector is a vector for outputting speech text; and a first determination subunit, configured to determine the first vector as the first speech code of the population in the speech dimension.

[0141] In some embodiments, the second output unit includes: a classification detection subunit, configured to perform behavior-dimensional classification detection on images in the video sample set using the behavior detection subnetwork to obtain a second vector, wherein the second vector is a vector used to output a classification score; and a second determination subunit, configured to determine the second vector as a first behavior code of the crowd in the behavior dimension.

[0142] In some embodiments, the third output unit includes: a target tracking and detection subunit, configured to perform target tracking and detection in the motion trajectory dimension on images in the video sample set using the target tracking and detection subnetwork, to obtain a third vector and a target detection box, wherein the third vector is a vector used to output the target detection box; a transformation subunit, configured to obtain the center point coordinates of the target detection box and transform the center point coordinates into a fourth vector; and a splicing subunit, configured to determine the spliced ​​third vector and the fourth vector as the first motion trajectory code of the crowd in the motion trajectory dimension.

[0143] In some embodiments, the apparatus further includes: a second acquisition module, configured to acquire the classification vector output in the classification sub-model; a clustering module, configured to perform clustering processing on the classification vector to obtain the updated categories of people in the crowd; and an annotation module, configured to re-annotate the speech sample set and the video sample set based on the updated categories.

[0144] In some embodiments, the updated categories include: shop assistants, first-level interested customers, second-level interested customers, and third-level interested customers. Correspondingly, the clustering module includes: a clustering submodule, used to perform clustering processing on the classification vector to obtain at least one person category feature, wherein the person category feature is the feature corresponding to the cluster center obtained after clustering processing; and a determination submodule, used to determine, based on the at least one person category feature, the updated categories of people in the population, including: shop assistants, first-level interested customers, second-level interested customers, and third-level interested customers.

[0145] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0146] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0147] It should be noted that, in the embodiments of this application, if the above-mentioned person classification method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0148] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0149] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0150] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0151] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0152] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0153] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0154] Processor 701 typically controls the overall operation of computer device 700.

[0155] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0156] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0157] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0158] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0160] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0161] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0162] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0163] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0164] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method of classifying a person, characterized by, include: Acquire the voice and video data of the person to be detected; Based on the voice data and the video data, the first multimodal fusion code of the person to be detected in the three dimensions of voice, behavior and movement trajectory is output using the fusion sub-model in the crowd classification model; The feature extraction sub-model in the crowd classification model is used to extract features from the first multimodal fusion encoding to obtain the features of the person to be detected; Determine the distance between the features of the person to be detected and each person category feature in the person category feature set; wherein, the person category feature is the feature obtained by clustering the classification vector output by the classification sub-model in the crowd classification model after training the crowd classification model with the training sample set, and the classification vector is the vector used to output the classification score; Based on the distance, the target person category features are determined in the set of person category features; The category of the person corresponding to the target person category feature is determined as the category of the person to be detected; The training sample set includes a speech sample set and a video sample set. The crowd classification model is trained using the following methods: Based on the acquired set of voice samples and the set of video samples with labeled crowd categories, the second multimodal fusion code of the crowd in three dimensions—voice, behavior, and movement trajectory—is output using the fusion sub-model. The feature extraction sub-model is used to extract features from the second multimodal fusion encoding to obtain the features of individuals in the crowd; The classification sub-model is used to classify the characteristics of individuals in the population to obtain classification results; Based on the classification results and the labeled population category tags, the loss function of the population classification model is determined; Based on the loss function, train the population classification model until convergence; The fusion sub-model includes an encoding module and a fusion module. Based on the acquired set of voice samples and the set of video samples labeled with audience category tags, the fusion sub-model outputs a second multimodal fusion code for the audience across three dimensions: voice, behavior, and movement trajectory. This includes: Based on the acquired set of voice samples and the set of video samples with labeled population categories, the encoding module outputs the first voice code of the population in the voice dimension, the first behavior code in the behavior dimension, and the first movement trajectory code in the movement trajectory dimension. The fusion module is used to convert the first speech code, the first behavior code, and the first movement trajectory code into the same implicit space to obtain the second speech code, the second behavior code, and the second movement trajectory code. The second speech code, the second behavior code, and the second movement trajectory code are concatenated to obtain the second multimodal fusion code.

2. The method of claim 1, wherein, The encoding module includes: a speech feature extraction subnetwork, a behavior detection subnetwork, and a target tracking and detection subnetwork. Based on the acquired set of speech samples labeled with crowd category tags and the set of video samples, the encoding module outputs a first speech code in the speech dimension, a first behavior code in the behavior dimension, and a first movement trajectory code in the movement trajectory dimension for the crowd. This includes: Based on the set of speech samples of the population, the speech feature extraction subnetwork is used to output the first speech code of the population in the speech dimension; Based on the video sample set of the population, the behavior detection subnetwork is used to output the first behavior code of the population in the behavior dimension; Based on the video sample set of the crowd, the target tracking and detection subnetwork is used to output the first movement trajectory code of the crowd in the movement trajectory dimension.

3. The method of claim 2, wherein, The speech sample set based on the population, using the speech feature extraction sub-network, outputs the first speech code of the population in the speech dimension, including: The speech feature extraction subnetwork is used to extract speech features from the speech sample set to obtain a first vector, wherein the first vector is a vector used to output speech text; The first vector is determined as the first speech code of the population in the speech dimension.

4. The method of claim 2, wherein, The video sample set based on the population, using the behavior detection sub-network, outputs the first behavior code of the population in the behavior dimension, including: The behavior detection subnetwork is used to classify and detect the images in the video sample set according to the behavior dimension, and a second vector is obtained, wherein the second vector is a vector used to output the classification score; The second vector is determined as the first behavioral code of the population in the behavioral dimension.

5. The method of claim 2, wherein, The video sample set based on the crowd, using the target tracking and detection subnetwork, outputs the first movement trajectory code of the crowd in the movement trajectory dimension, including: The target tracking and detection subnetwork is used to perform target tracking and detection in the motion trajectory dimension of the images in the video sample set to obtain a third vector and a target detection box, wherein the third vector is the vector used to output the target detection box; Obtain the center point coordinates of the target detection box, and convert the center point coordinates into a fourth vector; The concatenated third and fourth vectors are determined as the first movement trajectory code of the population in the movement trajectory dimension.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: Obtain the classification vector output from the classification sub-model; Clustering is performed on the classification vectors to obtain the updated categories of people in the population; Based on the updated categories, the audio sample set and the video sample set are re-labeled.

7. The method according to claim 6, characterized in that, The updated categories include: store clerks, first-level interested customers, second-level interested customers, and third-level interested customers. Correspondingly, the clustering process performed on the classification vector to obtain the updated categories of individuals in the population includes: The classification vector is clustered to obtain at least one person category feature, wherein the person category feature is the feature corresponding to the cluster center obtained after the clustering process; Based on the at least one person category feature, the updated categories of people in the population are determined to include: shop assistants, first-level interested customers, second-level interested customers, and third-level interested customers.

8. A person classification device, characterized in that, include: The first acquisition module is used to acquire the voice data and video data of the person to be detected; The first output module is used to output the first multimodal fusion code of the person to be detected in three dimensions: voice, behavior and movement trajectory, based on the voice data and the video data and using the fusion sub-model in the crowd classification model. The first extraction module is used to extract features from the first multimodal fusion encoding using the feature extraction sub-model in the crowd classification model to obtain the features of the person to be detected. The first determining module is used to determine the distance between the features of the person to be detected and each person category feature in the person category feature set; wherein, the person category feature is the feature obtained by clustering the classification vector output by the classification sub-model in the crowd classification model after training the crowd classification model with the training sample set, and the classification vector is the vector used to output the classification score; the training sample set includes a voice sample set and a video sample set; The second determining module is used to determine the target person category feature in the person category feature set based on the distance; The third determining module is used to determine the category of the person corresponding to the target person category feature as the category of the person to be detected; The second output module is used to output the second multimodal fusion code of the crowd in three dimensions—voice, behavior, and movement trajectory—based on the acquired set of voice samples and the set of video samples with labeled crowd category tags, using the fusion sub-model. The second extraction module is used to extract features from the second multimodal fusion encoding using the feature extraction sub-model to obtain the features of the people in the crowd; The classification module is used to classify the features of individuals in the population using the classification sub-model, and obtain the classification result; The fourth determining module is used to determine the loss function of the population classification model based on the classification results and the labeled population category labels; The training module is used to train the population classification model to convergence based on the loss function. The fusion sub-model includes an encoding module and a fusion module. The second output module includes: an output sub-module, used to output, based on the acquired set of voice samples labeled with audience category tags and the set of video samples, the first voice code, the first behavior code, and the first movement trajectory code of the audience in the voice dimension, using the encoding module; a transformation sub-module, used to transform the first voice code, the first behavior code, and the first movement trajectory code into the same implicit space using the fusion module, to obtain a second voice code, a second behavior code, and a second movement trajectory code; and a splicing sub-module, used to splice the second voice code, the second behavior code, and the second movement trajectory code to obtain the second multimodal fusion code.

Images