Construction method, device and equipment of intelligent public security digital person and storage medium

Abstract

The invention discloses an intelligent public security digital person construction method and device, equipment and a storage medium, and the method comprises the steps: obtaining the motion sample data of the body feature point location of a sampling person in a sampling process, generating a character feature set according to the motion sample data, receiving the interaction information of an interaction object, carrying out the feature separation of the interaction information, and obtaining a character feature set; generating a to-be-processed task according to the content feature, generating a response emotion according to the emotion feature, and performing association matching on the to-be-processed task based on the police affair knowledge base to obtain response information corresponding to the to-be-processed task; matching the limb action set and the face action set according to the response information and the response emotion to generate a digital human image frame, obtaining a digital human response animation and a digital human response audio based on time sequence information of the digital human image frame, and constructing an intelligent public security digital human image according to a digital human image material library and the digital human response animation. And outputting the digital human response audio. The digital person for processing police affairs is constructed in the mode.

Description

Technical Field

[0001] The present application relates to the technical field of virtual digital humans, and in particular to methods, devices, equipment and storage media for constructing smart public security digital humans. Background Art

[0002] With the rapid development of information technology, the public security system faces new challenges in maintaining social stability, combating crime and serving the people's livelihood. The traditional police work model mainly relies on manpower and limited technical means. The information between departments is not interoperable, resulting in the inability to effectively share and utilize data. In addition, when public emergencies occur, the traditional manual dispatch method is inefficient and difficult to respond quickly. The processing capacity of massive unstructured data such as video and audio is limited, and it is difficult to extract valuable information from it to prevent and combat crime. Therefore, the current electronic police system cannot meet the current needs of police activities, and there are technical problems such as low data sharing and low data processing efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a method, device, equipment and storage medium for constructing a smart public security digital human, aiming to solve the technical problems of low data sharing and low data processing efficiency in the prior art.

[0005] To achieve the above purpose, this application provides a method for constructing a smart public security digital human, the method comprising: Acquire action sample data of body feature points of the sampling personnel during the sampling process, and generate a character feature set according to the action sample data, wherein the character feature set includes a body action set and a facial action set; Receiving interactive information of an interactive object, performing feature separation on the interactive information to obtain content features and emotional features, generating a task to be processed according to the content features, and generating a response emotion according to the emotional features; Performing correlation matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks; Generating a digital human image frame according to the response information and the response emotion matching the body action set and the facial action set, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame; The image of a smart public security digital human is constructed according to the digital human image library and the digital human response animation, and the digital human response audio is output.

[0006] In one embodiment, the step of obtaining the action sample data of the body feature points of the sampled person during the sampling process and generating the character feature set according to the action sample data comprises: Acquire image data and sensor data of sampling personnel during the sampling process; Matching the image data and the sensor data with body feature points respectively to obtain image features and sensor features of each body point; Determine a sampling action in the sampling process, map the sampling action with the image feature and the sensor feature, and obtain action sample data; The action sample data are grouped according to the sampled actions to obtain a character feature set.

[0007] In one embodiment, the steps of receiving interaction information of an interaction object, performing feature separation on the interaction information to obtain content features and emotion features, generating a task to be processed according to the content features, and generating a response emotion according to the emotion features include: Receiving interactive information of an interactive object, and performing feature separation on the interactive information to obtain text information and non-text information; Performing text extraction on the non-text information to obtain hidden text information and audio information in the non-text information; Segmenting the text information and the hidden text information to determine key text elements; Sorting the key text elements by features to obtain content features; Performing emotion extraction on the audio information, determining audio pitch changes and audio volume changes, and obtaining emotion features according to the audio pitch changes and the audio volume changes; A task to be processed is generated according to the content features, and a response emotion is generated according to the emotion features.

[0008] In one embodiment, before the step of performing association matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks, the step further includes: Accessing a police database, copying the readable data in the police database, and obtaining police sample data; Performing data cleaning on the police sample data to obtain sample cleaned data; Determine a data association mapping of the sample cleaned data, and perform feature marking on the sample cleaned data to obtain a marking mapping; Extracting key data features of the sample cleaned data, and determining the key data features as retrieval tags; According to the combination of the retrieval tag and the tag mapping, a police knowledge base is obtained.

[0009] In one embodiment, the step of performing association matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks includes: Determine the entities to be processed in the tasks to be processed, and divide the entities to be processed into main entities and auxiliary entities, match the main entities with source entities in the police knowledge base, and when the match is successful, determine the source entity as the target entity; Determine a relationship mapping of the target entity, and use the relationship mapping including the auxiliary entity as a candidate relationship mapping; Performing relevance evaluation on the candidate relationship mappings to obtain a relevance score, and taking the candidate relationship mapping corresponding to the highest relevance score as the target relationship mapping; Generate response information according to the target relationship mapping.

[0010] In one embodiment, the steps of matching the body action set and the facial action set to generate a digital human image frame according to the response information and the response emotion, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame include: Segmenting the response information into a character set, matching the character set with the facial action set character by character, and determining the lip features corresponding to each character; Determine the body action timestamp according to the action matching word in the response information; Matching the body movement set according to the response emotion to obtain a target body movement; Combining the lip feature with the target body movement according to the timestamp to obtain a digital human image frame; Arranging the response animation frames in time sequence to obtain a digital human response animation; A digital human response audio is generated according to the response information, the preset timbre and the response emotion.

[0011] In one embodiment, the step of constructing a smart public security digital human image according to the digital human image library and the digital human response animation, and outputting the digital human response audio comprises: Determine the current interaction time and image setting parameters, and determine the appearance of the target digital human in the digital human image material library according to the interaction time and the image setting parameters; The target digital human appearance is integrated with the digital human response animation to construct a smart public security digital human image, and the digital human response audio is output.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a construction device for a smart police digital human, which comprises: A character sampling module is used to obtain action sample data of the body feature points of the sampled person during the sampling process, and generate a character feature set according to the action sample data, wherein the character feature set includes a body action set and a facial action set; A task interaction module is used to receive interaction information of an interaction object, perform feature separation on the interaction information to obtain content features and emotional features, generate a task to be processed according to the content features, and generate a response emotion according to the emotional features; A task processing module, used for performing correlation matching on the tasks to be processed based on the police knowledge base, and obtaining response information corresponding to the tasks to be processed; A task response module, used to generate a digital human image frame according to the response information and the response emotion matching the body action set and the facial action set, and obtain a digital human response animation and a digital human response audio based on the timing information of the digital human image frame; The digital human response module is used to construct a smart public security digital human image according to the digital human image library and the digital human response animation, and output the digital human response audio.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a construction device for a smart public security digital human, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the construction method for a smart public security digital human as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for constructing a smart public security digital human as described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the method for constructing a smart public security digital human as described above are implemented.

[0016] The present application provides a method for constructing a smart public security digital human, by obtaining the action sample data of the sampling personnel's body feature points during the sampling process, generating a character feature set based on the action sample data, receiving the interactive information of the interactive object, performing feature separation on the interactive information, obtaining content features and emotional features, generating tasks to be processed based on the content features, generating response emotions based on the emotional features, performing correlation matching on the tasks to be processed based on the police knowledge base, obtaining response information corresponding to the tasks to be processed, matching the body action set and the facial action set based on the response information and the response emotion to generate a digital human image frame, obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame, constructing a smart public security digital human image based on the digital human image material library and the digital human response animation, and outputting the digital human response audio. By constructing a digital human for handling police affairs in the above manner, by the above method, a digital human with real human action behaviors, capable of interacting with interactive objects, and providing corresponding response information based on the interactive objects and interactive information can be constructed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 This is a flow chart of Embodiment 1 of the method for constructing a smart police digital human in this application; Figure 2 This is a schematic diagram of a full-body digital human sampling model according to an embodiment of the method for constructing a smart public security digital human of this application; Figure 3 This is a schematic diagram of facial action feature recognition point distribution in an embodiment of the method for constructing a smart police digital human in this application; Figure 4 A schematic diagram of a simplified process of constructing a smart police digital human provided in Example 2 of this application; Figure 5 This is a schematic diagram of the module structure of a construction device for a smart police digital human according to an embodiment of the present application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the method for constructing a smart public security digital human in the embodiment of the present application.

[0020] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: obtaining the action sample data of the body feature points of the sampling personnel during the sampling process, generating a character feature set according to the action sample data, and the character feature set includes a body action set and a facial action set; receiving the interaction information of the interactive object, performing feature separation on the interaction information to obtain content features and emotional features, generating a task to be processed according to the content features, and generating a response emotion according to the emotional features; performing association matching on the task to be processed based on the police knowledge base to obtain response information corresponding to the task to be processed; matching the body action set and the facial action set according to the response information and the response emotion to generate a digital human image frame, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame; constructing a smart public security digital human image according to the digital human image material library and the digital human response animation, and outputting the digital human response audio.

[0024] At present, with the rapid development of information technology, the public security system faces new challenges in maintaining social stability, combating crime and serving the people's livelihood. The traditional police work model mainly relies on manpower and limited technical means. The information between departments is not interoperable, resulting in the inability to effectively share and utilize data. In addition, when public emergencies occur, the traditional manual dispatch method is inefficient and difficult to respond quickly. The processing capacity of massive unstructured data such as video and audio is limited, and it is difficult to extract valuable information from it to prevent and combat crime. Therefore, the current electronic police system cannot meet the current needs of police activities, and there are technical problems such as low data sharing and low data processing efficiency.

[0025] The present application provides a solution, by obtaining the action sample data of the sampling personnel's body feature points during the sampling process, generating a character feature set based on the action sample data, receiving the interactive information of the interactive object, separating the features of the interactive information, obtaining content features and emotional features, generating tasks to be processed based on the content features, generating response emotions based on the emotional features, performing correlation matching on the tasks to be processed based on the police knowledge base, obtaining response information corresponding to the tasks to be processed, matching the body action set and the facial action set based on the response information and the response emotion to generate a digital human image frame, obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame, constructing a smart public security digital human image based on the digital human image material library and the digital human response animation, and outputting the digital human response audio. By constructing a digital human for handling police affairs in the above manner, by the above method, a digital human with real human action behaviors, capable of interacting with interactive objects, and providing corresponding response information based on the interactive objects and interactive information can be constructed.

[0026] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, a construction device for a smart police digital human, etc., and this embodiment does not specifically limit this. The following takes the construction device for a smart police digital human as an example to illustrate this embodiment and the following embodiments.

[0027] The present application embodiment provides a method for constructing a smart public security digital human. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for constructing a smart police digital human in this application.

[0028] In this embodiment, the method for constructing a smart police digital human includes steps S10 to S50: Step S10, obtaining motion sample data of the body feature points of the sampled personnel during the sampling process, and generating a character feature set according to the motion sample data, wherein the character feature set includes a body motion set and a facial motion set.

[0029] It should be noted that the sampling personnel are the personnel who provide behavioral features for the digital human. Multiple feature point collectors are set on the sampling personnel's body to obtain the facial or limb motion features on the body of the sampling personnel when they are performing actions, including the displacement data of the feature points. The facial and limb motion features obtained constitute the character feature set, which includes a limb motion set and a facial motion set.

[0030] In a specific implementation, according to the construction type of the digital human, a half-body digital human and a full-body digital human can be constructed. The number of feature points of a half-body digital human at the same feature density is less than that of a full-body digital human. Therefore, the construction type of the digital human can be selected according to the specific situation, and this embodiment does not limit it. Figure 2 , Figure 2 This is a schematic diagram of the full-body digital human sampling model. Motion capture devices can be installed at various positions of the sampling personnel that can respond to movements, and are used to obtain movement data, and to perform full-body modeling on the sampling personnel to obtain the facial movement features of the sampling personnel. Figure 3 , Figure 3 This is a schematic diagram of the distribution of facial action feature recognition points. The figure takes 40, 60, 80, and 100 feature point diagrams as examples. It can be seen that when the facial area is the same, the more feature points there are on the face, the more feature points will reflect facial actions, and the fitted facial expression will be closer to the real facial expression. Therefore, there is a positive correlation between the number of facial feature points and the facial action sampling effect. After determining the number of facial feature points, the facial feature points can be numbered sequentially and a time series matrix for each feature point can be generated. , the eigenvalues ​​in the time series matrix at the same time t are combined into a facial action sequence to obtain the facial action features at time t. Similarly, the construction method of the body action feature points is similar to that of the facial action features, which will not be described in detail in this embodiment.

[0031] In a feasible implementation manner, the step of obtaining the action sample data of the body feature points of the sampled person during the sampling process and generating a character feature set according to the action sample data comprises: Acquire image data and sensor data of sampling personnel during the sampling process; Matching the image data and the sensor data with body feature points respectively to obtain image features and sensor features of each body point; Determine a sampling action in the sampling process, map the sampling action with the image feature and the sensor feature, and obtain action sample data; The action sample data are grouped according to the sampled actions to obtain a character feature set.

[0032] It should be noted that the image data is obtained by obtaining the video stream of the sampling person during the sampling process through a camera (which can be an RGB camera or a depth camera); the sensor data is captured by devices such as IMUs (inertial measurement units), pressure sensors, and electromyography (EMG) to capture limb movements and muscle activities.

[0033] In a specific implementation, the image data and sensor data of the camera and the sensor device are obtained, wherein the image data can be in the form of image frames. In the form of Represents the image frame at time t. Represents the data vector from n sensors at time t, matches these data with predefined body feature points, and assumes is the position coordinate of the i-th feature point at time t, is the image feature corresponding to the feature point, and It is the feature extracted from the sensor data. Image features , f is the mapping function, which can transform the image and location Mapped to feature space. Sensor features , g is a function that can be obtained from sensor data Extract and feature points Related information. is the kth sampling action, then for each , we can get a set of feature pairs . The mapping relationship can be expressed as:

[0034] Then the action sample data is grouped according to different collection actions to obtain character feature sets. contains a set of feature vectors, expressed as:

[0035] Among them, m is the total number of sampled actions, each It is a feature matrix whose rows correspond to different time points and columns contain the image features and sensor features of each feature point.

[0036] Step S20, receiving interaction information of the interaction object, performing feature separation on the interaction information to obtain content features and emotion features, generating tasks to be processed according to the content features, and generating response emotions according to the emotion features.

[0037] It should be noted that the interactive object is the object that communicates with the digital human, where the interactive object can be a person or other intelligent device, etc.; the interactive information can be data in various forms including text, audio, pictures, etc.

[0038] It should be understood that content features refer to features in interactive information used to describe specific processing matters, and emotional features refer to audio features such as tone and intonation or text features in interactive information.

[0039] In the specific implementation, first, the system needs to receive input information from the interactive object. The input information can be text, voice or other forms of input. For text input, it can be processed directly; for voice input, it needs to be converted into text through automatic speech recognition (ASR) technology. Then, the input interactive information uses algorithms such as TF-IDF and TextRank to extract keywords from the text, identify and classify named entities in the text, such as names of people, places, and organizations, and predict the user's intention through a trained machine learning model or deep learning model. Emotional features refer to the emotional tendency expressed in the interactive information. The text is scored using an emotional dictionary to determine the degree of positive or negative emotions, or a pre-trained language model is fine-tuned to adapt to specific sentiment analysis tasks. Based on the content features, the user's needs or requests can be determined and the corresponding tasks to be processed can be generated. Based on the emotional features, the response method can be adjusted to better match the user's emotions. This can be achieved by selecting an appropriate reply template, adjusting the tone, or using empathy. Finally, the generated tasks and responses are integrated together to form the final output of the system, including task execution results and emotional replies.

[0040] In a feasible implementation manner, the steps of receiving interaction information of an interaction object, performing feature separation on the interaction information to obtain content features and emotional features, generating tasks to be processed according to the content features, and generating response emotions according to the emotional features include: Receiving interactive information of an interactive object, and performing feature separation on the interactive information to obtain text information and non-text information; Performing text extraction on the non-text information to obtain hidden text information and audio information in the non-text information; Segmenting the text information and the hidden text information to determine key text elements; Sorting the key text elements by features to obtain content features; Performing emotion extraction on the audio information, determining audio pitch changes and audio volume changes, and obtaining emotion features according to the audio pitch changes and the audio volume changes; A task to be processed is generated according to the content features, and a response emotion is generated according to the emotion features.

[0041] In the specific implementation, the interactive information may include text information (such as chat records) and non-text information (such as voice, image). First, it is necessary to receive and preliminarily classify this information. Let I be the received interactive information, where: T represents text information, V represents voice data, I represents mis image data. Then, feature separation and feature extraction are performed on text information, voice data, and image data respectively, and they are converted into text using automatic speech recognition (ASR) technology, and hidden text information H and audio information A are extracted from them. The text information T and hidden text information H are segmented to obtain the word sequence W. T and W H , using natural language processing technology to determine key text elements from word sequences. The specific formula is:

[0042] in, is the word frequency, is the inverse document frequency.

[0043] When extracting emotional features, the pitch change and volume change can be used as indicators of emotional features according to the formula:

[0044]

[0045] Among them, f and g are functions that map pitch and audio intensity to emotional features.

[0046] The emotional characteristics are: .

[0047] When generating content features, the importance of features can be calculated using cosine similarity:

[0048] in, are the eigenvalues ​​of the two vectors.

[0049] Finally, it is possible to generate tasks to be processed based on content features and generate response emotions based on emotional features.

[0050] Step S30, performing association matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks.

[0051] It should be noted that the police knowledge base is a data set formed by all police information that can be queried and disclosed in the police system, and the response information is the result information obtained after processing the task to be processed based on the police knowledge base.

[0052] In the specific implementation, when performing association matching, NLP technology can be used to match entries in the police knowledge base. First, the entries in the police knowledge base can be converted into vector representations, and then candidate matches are determined by calculating the similarity between the task to be processed and the entries in the police knowledge base, and response information is generated based on the candidate matches.

[0053] In a feasible implementation manner, the step of performing association matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks includes: Determine the entities to be processed in the tasks to be processed, and divide the entities to be processed into main entities and auxiliary entities, match the main entities with source entities in the police knowledge base, and when the match is successful, determine the source entity as the target entity; Determine a relationship mapping of the target entity, and use the relationship mapping including the auxiliary entity as a candidate relationship mapping; Performing relevance evaluation on the candidate relationship mappings to obtain a relevance score, and taking the candidate relationship mapping corresponding to the highest relevance score as the target relationship mapping; Generate response information according to the target relationship mapping.

[0054] In the specific implementation, the entities to be processed in the tasks to be processed can be determined, and the entities to be processed can be divided into main entities and auxiliary entities. In the matching process, the main entity is matched first. When there are multiple matching results according to the main entity, the auxiliary entity is matched in the matching results to obtain a certain number of matching results, which are determined as the target entity. Then the similarity between the entity to be processed and the target entity is calculated according to the specific formula:

[0055] in, They are the vector representation of the entity to be processed and the vector representation of the target entity respectively.

[0056] Then calculate the Euclidean distance D between the entity to be processed and the target entity. . A mapping relationship is formed between the target entity and the entity to be processed using the Euclidean distance, and the similarity value is used as the relevance evaluation. At the same time, the similarity values ​​are sorted from large to small, and the candidate relationship mapping with the highest relevance score is used as the target mapping relationship, and then the response information is generated. Among the other mapping relationships, a certain number are selected as reference response information, and no active response is made, and an optional viewing option is provided.

[0057] In a feasible implementation manner, before the step of associating and matching the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks, the step further includes: Accessing a police database, copying the readable data in the police database, and obtaining police sample data; Performing data cleaning on the police sample data to obtain sample cleaned data; Determine a data association mapping of the sample cleaned data, and perform feature marking on the sample cleaned data to obtain a marking mapping; Extracting key data features of the sample cleaned data, and determining the key data features as retrieval tags; According to the combination of the retrieval tag and the tag mapping, a police knowledge base is obtained.

[0058] In the specific implementation, the construction of digital human needs to establish a connection with the police database, so it can access the police database and copy the readable data in the police database for examination to obtain the police sample data. Then the police sample data is cleaned, and the privacy-related data in the police sample data is modified to play a certain confidentiality role. In order to ensure the readability of the data, access rights can be set in the database, and the interactive objects can be authenticated. Different query contents are displayed according to different access rights to prevent the risk of data leakage. After the sample data is cleaned, the sample cleaned data can be obtained, so the data association mapping between the sample data can be determined. The sample cleaned data can be set into two tables, one of which is the source data and the other is the mapping data. The connection relationship between the source data and the mapping data can be expressed in the form of a vector, and the association information between the data can allow association, that is, A maps B, B maps C, then C can be considered as the association data of A. A specific example is that in case 1, there is a suspect with feature group X who commits crime Z, and in case 2, there is a suspect with feature group Y who commits crime Z. At this time, there is a relationship: X maps to Z, Y maps to Z, then association can be triggered, and X can be associated with Y. At this time, guidance can be given to compare the feature groups of X and Y. X, Y, and Z can all be identified as key data features as retrieval tags. As long as the interactive information includes the retrieval tag, the corresponding data can be locked. For the convenience of storage, the retrieval tag and tag mapping can be combined to obtain a police knowledge base.

[0059] Step S40, generating a digital human image frame according to the response information and the response emotion matching the body action set and the facial action set, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame.

[0060] It should be noted that the digital human image frame refers to the freeze-frame picture of the digital human's facial and body movements, the digital human's behavioral movements are obtained by continuously playing multiple consecutive digital human image frames, and the response audio is a sound synthesized based on text-to-speech and preset timbre.

[0061] In a specific implementation, the lip movement features of the lips corresponding to each character lock in the text can be determined according to the response information for matching, and the emotional features can be matched with the facial movements to obtain the facial movement part and the body movement part corresponding to the response data, wherein the facial movement is consistent with the time of the text. The digital human response audio is synthesized by combining the response information with the preset timbre, and the audio information is time-unified with the facial movement features and body movement features to obtain the digital human response animation and digital human response audio.

[0062] In a feasible implementation manner, the steps of matching the body action set and the facial action set to generate a digital human image frame according to the response information and the response emotion, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame include: Segmenting the response information into a character set, matching the character set with the facial action set character by character, and determining the lip features corresponding to each character; Determine the body action timestamp according to the action matching word in the response information; Matching the body movement set according to the response emotion to obtain a target body movement; Combining the lip feature with the target body movement according to the timestamp to obtain a digital human image frame; Arranging the response animation frames in time sequence to obtain a digital human response animation; A digital human response audio is generated according to the response information, the preset timbre and the response emotion.

[0063] In the specific implementation, the response information Split into character set C, each character Corresponding to a time step , , n is the number of characters, is the i-th character. For each character , from a pre-built facial action set Select the matching lip features from . You can use pre-trained lip movement models to achieve this matching. ,set up is the set of lip features corresponding to all characters.

[0064] According to the response information Match the action words in the text box to determine the timestamp of the body action. ,set up In response to the keywords in the message, is the timestamp of the jth keyword, then According to the response emotion E, The target body movements are matched, and the obtained lip features are combined with the target body movements to obtain the digital human image frame at the current timestamp. In order to obtain the digital human response animation, the response animation frames can be arranged in time sequence. At this time, the image features of the digital human have been obtained. In order to achieve voice communication, the response information can be combined with the preset timbre and the response emotion to generate the digital human response audio.

[0065] Step S50: constructing a smart public security digital human image according to the digital human image library and the digital human response animation, and outputting the digital human response audio.

[0066] It should be noted that the digital human image material library refers to the image material that constitutes the digital human image, which is composed of multiple texture maps. The digital human image can be rendered to a certain extent after combining multiple texture maps.

[0067] In the specific implementation, according to the current interaction time or weather information, the matching clothing can be determined from the digital human image library, and the image setting can be realized based on the interaction time. At the same time, the digital human can also set specific image parameters for the external display of the digital human. After the image of the digital human is determined, the digital human can be rendered to make the digital human image more realistic. At the same time, after the image of the digital human is determined, the digital human response audio can be output to complete the digital human image and response settings. The behavior of being able to communicate and ask questions with the smart digital human is realized, and the smart digital human can respond accordingly based on the communication questions. It can also assist police officers in quickly searching files, determining the similarities between multiple files, etc., to improve the efficiency of police officers in handling cases.

[0068] This embodiment provides a method for constructing a smart police digital human, by obtaining the action sample data of the body feature points of the sampling personnel during the sampling process, generating a character feature set according to the action sample data, receiving the interactive information of the interactive object, performing feature separation on the interactive information, obtaining content features and emotional features, generating a task to be processed according to the content features, generating a response emotion according to the emotional features, performing association matching on the task to be processed based on the police knowledge base, obtaining response information corresponding to the task to be processed, matching the body action set and the facial action set according to the response information and the response emotion to generate a digital human image frame, obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame, constructing a smart police digital human image according to the digital human image material library and the digital human response animation, and outputting the digital human response audio. By constructing a digital human for handling police affairs in the above manner, by the above method, a digital human with real human action behaviors, capable of interacting with interactive objects, and providing corresponding response information based on the interactive objects and interactive information can be constructed.

[0069] For example, to help understand the implementation process of the method for constructing a smart police digital human obtained by combining this embodiment with the above-mentioned embodiment 2, please refer to Figure 4 , Figure 4 A brief flow diagram of a method for constructing a smart public security digital human is provided. Specifically, when constructing a digital human, the appearance of the digital human can be constructed by combining the image material library of the digital human and the action characteristics of the digital human, and the voice image of the digital human can be determined by the timbre characteristics of the digital human. The appearance image and the voice image can be combined to construct a smart digital human. The smart digital human can generate response information according to the input interactive information, and then match the action characteristics and the voice image according to the response information respectively to obtain output content, respond in the image of the smart digital human, and realize information communication with the smart digital human.

[0070] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the construction method of the smart public security digital human of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0071] This application also provides a construction device for a smart public security digital human. Please refer to Figure 5 , the construction device of the smart public security digital human includes: The character sampling module 10 is used to obtain the action sample data of the body feature points of the sampled person during the sampling process, and generate a character feature set according to the action sample data, wherein the character feature set includes a body action set and a facial action set.

[0072] The task interaction module 20 is used to receive interaction information of an interaction object, perform feature separation on the interaction information to obtain content features and emotion features, generate a task to be processed according to the content features, and generate a response emotion according to the emotion features.

[0073] The task processing module 30 is used to perform correlation matching on the tasks to be processed based on the police knowledge base to obtain response information corresponding to the tasks to be processed.

[0074] The task response module 40 is used to generate a digital human image frame according to the response information and the response emotion matching the body action set and the facial action set, and obtain a digital human response animation and a digital human response audio based on the timing information of the digital human image frame.

[0075] The digital human response module 50 is used to construct a smart public security digital human image according to the digital human image library and the digital human response animation, and output the digital human response audio.

[0076] In a feasible implementation, the character sampling module 10 is also used to obtain image data and sensor data of the sampling personnel during the sampling process; match the image data and the sensor data with body feature points respectively to obtain image features and sensor features of each body point; determine the sampling action in the sampling process, map the sampling action with the image features and the sensor features to obtain action sample data; group the action sample data according to the sampling action to obtain a character feature set.

[0077] In a feasible implementation manner, the task interaction module 20 is also used to receive interaction information of an interaction object, perform feature separation on the interaction information to obtain text information and non-text information; perform text extraction on the non-text information to obtain hidden text information and audio information in the non-text information; perform word segmentation on the text information and the hidden text information to determine key text elements; perform feature sorting on the key text elements to obtain content features; perform emotion extraction on the audio information to determine audio pitch changes and audio volume changes, and obtain emotion features based on the audio pitch changes and the audio volume changes; generate tasks to be processed based on the content features, and generate response emotions based on the emotion features.

[0078] In a feasible implementation, the task processing module 30 is also used to access the police database, copy the readable data in the police database to obtain police sample data; perform data cleaning on the police sample data to obtain sample cleaned data; determine the data association mapping of the sample cleaned data, and perform feature tagging on the sample cleaned data to obtain a tag mapping; extract key data features of the sample cleaned data, and determine the key data features as retrieval tags; and obtain a police knowledge base based on the combination of the retrieval tag and the tag mapping.

[0079] In a feasible implementation manner, the task processing module 30 is also used to determine the entities to be processed in the tasks to be processed, and divide the entities to be processed into main entities and auxiliary entities, match the main entity with the source entity in the police knowledge base, and when the match is successful, determine the source entity as the target entity; determine the relationship mapping of the target entity, and use the relationship mapping containing the auxiliary entity as a candidate relationship mapping; perform relevance evaluation on the candidate relationship mapping to obtain a relevance score, and use the candidate relationship mapping corresponding to the highest relevance score as the target relationship mapping; and generate response information according to the target relationship mapping.

[0080] In a feasible implementation manner, the task response module 40 is further used to divide the response information into a character set, match the character set with the facial action set character by character, and determine the lip features corresponding to each character; determine the limb action timestamp according to the action matching words in the response information; match the limb action set according to the response emotion to obtain the target limb action; combine the lip features with the target limb action according to the timestamp to obtain a digital human image frame; arrange the response animation frames in time sequence to obtain a digital human response animation; generate a digital human response audio according to the response information, a preset timbre and the response emotion.

[0081] In a feasible implementation manner, the digital human response module 50 is also used to determine the current interaction time and image setting parameters, determine the target digital human appearance in the digital human image library according to the interaction time and the image setting parameters; merge the target digital human appearance with the digital human response animation to construct a smart public security digital human image, and output the digital human response audio.

[0082] The construction device of the smart police digital human provided by the present application adopts the construction method of the smart police digital human in the above-mentioned embodiment, which can solve the technical problems of low data sharing degree and low data processing efficiency. Compared with the prior art, the beneficial effects of the construction device of the smart police digital human provided by the present application are the same as the beneficial effects of the construction method of the smart police digital human provided by the above-mentioned embodiment, and other technical features in the construction device of the smart police digital human are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0083] The present application provides a device for constructing a smart public security digital human, the device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method for constructing a smart public security digital human in the first embodiment.

[0084] Reference below Figure 6, which shows a schematic diagram of the structure of a construction device for a smart police digital human suitable for implementing the embodiment of the present application. The construction device for a smart police digital human in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The construction device of the smart public security digital human shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0085] like Figure 6 As shown, the construction device of the smart police digital human may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the construction device of the smart police digital human are also stored. The processing device 1001, the ROM 1002 and the RAM 1004 are connected to each other through the bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the construction device of the smart police digital human to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows the construction device of the smart police digital human with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0086] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0087] The construction device of the smart police digital human provided by the present application adopts the construction method of the smart police digital human in the above-mentioned embodiment, which can solve the technical problem of the construction of the smart police digital human. Compared with the prior art, the beneficial effects of the construction device of the smart police digital human provided by the present application are the same as the beneficial effects of the construction method of the smart police digital human provided by the above-mentioned embodiment, and the other technical features in the construction device of the smart police digital human are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0088] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0089] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0090] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for constructing a smart public security digital human in the above-mentioned embodiment.

[0091] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0092] The computer-readable storage medium may be included in the construction device of the smart police digital human; or may exist independently without being assembled into the construction device of the smart police digital human.

[0093] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the construction device of the smart police digital human, the construction device of the smart police digital human: obtains the action sample data of the body feature points of the sampling personnel during the sampling process, generates a character feature set according to the action sample data, and the character feature set includes a body action set and a facial action set; receives the interaction information of the interactive object, performs feature separation on the interaction information to obtain content features and emotional features, generates a task to be processed according to the content features, and generates a response emotion according to the emotional features; performs association matching on the task to be processed based on the police knowledge base to obtain response information corresponding to the task to be processed; matches the body action set and the facial action set according to the response information and the response emotion to generate a digital human image frame, and obtains a digital human response animation and a digital human response audio based on the timing information of the digital human image frame; constructs the image of the smart police digital human according to the digital human image material library and the digital human response animation, and outputs the digital human response audio.

[0094] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0097] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for constructing a smart police digital human, and can solve the technical problem of constructing a smart police digital human. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the method for constructing a smart police digital human provided in the above-mentioned embodiment, and will not be elaborated here.

[0098] The present application also provides a computer program product, including a computer program, which implements the steps of the method for constructing a smart public security digital human as described above when the computer program is executed by a processor.

[0099] The computer program product provided in this application can solve the technical problem of constructing a smart police digital human. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the method for constructing a smart police digital human provided in the above embodiment, and will not be elaborated here.

[0100] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for constructing a smart public security digital human, characterized in that: The method for constructing the smart public security digital human includes: Acquire action sample data of body feature points of the sampling personnel during the sampling process, and generate a character feature set according to the action sample data, wherein the character feature set includes a body action set and a facial action set; Receiving interactive information of an interactive object, performing feature separation on the interactive information to obtain content features and emotional features, generating a task to be processed according to the content features, and generating a response emotion according to the emotional features; Performing correlation matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks; Generating a digital human image frame according to the response information and the response emotion matching the body action set and the facial action set, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame; The image of a smart public security digital human is constructed according to the digital human image library and the digital human response animation, and the digital human response audio is output.

2. The method according to claim 1, characterized in that The step of obtaining the action sample data of the body feature points of the sampled person during the sampling process and generating a character feature set according to the action sample data comprises: Acquire image data and sensor data of sampling personnel during the sampling process; Matching the image data and the sensor data with body feature points respectively to obtain image features and sensor features of each body point; Determine a sampling action in the sampling process, map the sampling action with the image feature and the sensor feature, and obtain action sample data; The action sample data are grouped according to the sampled actions to obtain a character feature set.

3. The method according to claim 1, characterized in that The steps of receiving interaction information of an interaction object, performing feature separation on the interaction information to obtain content features and emotional features, generating tasks to be processed according to the content features, and generating response emotions according to the emotional features include: Receiving interactive information of an interactive object, and performing feature separation on the interactive information to obtain text information and non-text information; Performing text extraction on the non-text information to obtain hidden text information and audio information in the non-text information; Segmenting the text information and the hidden text information to determine key text elements; Sorting the key text elements by features to obtain content features; Performing emotion extraction on the audio information, determining audio pitch changes and audio volume changes, and obtaining emotion features according to the audio pitch changes and the audio volume changes; A task to be processed is generated according to the content features, and a response emotion is generated according to the emotion features.

4. The method according to claim 1, characterized in that Before the step of associating and matching the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks, the method further includes: Accessing a police database, copying the readable data in the police database, and obtaining police sample data; Performing data cleaning on the police sample data to obtain sample cleaned data; Determine a data association mapping of the sample cleaned data, and perform feature marking on the sample cleaned data to obtain a marking mapping; Extracting key data features of the sample cleaned data, and determining the key data features as retrieval tags; According to the combination of the retrieval tag and the tag mapping, a police knowledge base is obtained.

5. The method according to claim 1, characterized in that The step of performing association matching on the pending tasks based on the police knowledge base to obtain response information corresponding to the pending tasks includes: Determine the entities to be processed in the tasks to be processed, and divide the entities to be processed into main entities and auxiliary entities, match the main entities with source entities in the police knowledge base, and when the match is successful, determine the source entity as the target entity; Determine a relationship mapping of the target entity, and use the relationship mapping including the auxiliary entity as a candidate relationship mapping; Performing relevance evaluation on the candidate relationship mappings to obtain a relevance score, and taking the candidate relationship mapping corresponding to the highest relevance score as the target relationship mapping; Generate response information according to the target relationship mapping.

6. The method according to claim 1, characterized in that The steps of matching the body action set and the facial action set to generate a digital human image frame according to the response information and the response emotion, and obtaining a digital human response animation and a digital human response audio based on the timing information of the digital human image frame include: Segmenting the response information into a character set, matching the character set with the facial action set character by character, and determining the lip features corresponding to each character; Determine the body action timestamp according to the action matching word in the response information; Matching the body movement set according to the response emotion to obtain a target body movement; Combining the lip feature with the target body movement according to the timestamp to obtain a digital human image frame; Arranging the digital human image frames in time sequence to obtain a digital human response animation; A digital human response audio is generated according to the response information, the preset timbre and the response emotion.

7. The method according to claim 1, characterized in that The steps of constructing a smart public security digital human image according to the digital human image library and the digital human response animation, and outputting the digital human response audio include: Determine the current interaction time and image setting parameters, and determine the appearance of the target digital human in the digital human image material library according to the interaction time and the image setting parameters; The target digital human appearance is integrated with the digital human response animation to construct a smart public security digital human image, and the digital human response audio is output.

8. A device for constructing a smart public security digital human, characterized in that: The construction device of the smart public security digital human includes: A character sampling module is used to obtain action sample data of the body feature points of the sampled person during the sampling process, and generate a character feature set according to the action sample data, wherein the character feature set includes a body action set and a facial action set; A task interaction module is used to receive interaction information of an interaction object, perform feature separation on the interaction information to obtain content features and emotional features, generate a task to be processed according to the content features, and generate a response emotion according to the emotional features; A task processing module, used for performing correlation matching on the tasks to be processed based on the police knowledge base, and obtaining response information corresponding to the tasks to be processed; A task response module, used to generate a digital human image frame according to the response information and the response emotion matching the body action set and the facial action set, and obtain a digital human response animation and a digital human response audio based on the timing information of the digital human image frame; The digital human response module is used to construct a smart public security digital human image according to the digital human image library and the digital human response animation, and output the digital human response audio.

9. A device for constructing a smart public security digital human, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for constructing a smart public security digital human as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for constructing a smart public security digital human as described in any one of claims 1 to 7 are implemented.

Images