Multimodal human-computer collaborative interaction system and method

Abstract

The present invention discloses a multimodal human-computer collaborative interaction system and method, which relates to the technical field of human-computer collaborative interaction, including: collecting surveillance videos shot by an intelligent question-answering robot with moving objects in the picture, extracting the target moving object and the human voice corresponding to the target moving object; obtaining the human voice speed corresponding to the human voice, obtaining the response text corresponding to the human voice, and obtaining the key coefficient corresponding to each keyword in the response text; obtaining the response speed of each keyword; adjusting the response speed between two adjacent keywords according to any two adjacent keywords, and then obtaining the response voice corresponding to the response text. The present invention analyzes the surveillance video corresponding to the intelligent question-answering robot, obtains the human voice corresponding to the moving object, and adjusts the robot's speech speed to better meet the needs of the user according to the human voice, which helps to improve the efficiency and fluency of human-computer collaborative interaction.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer collaborative interaction, and in particular to a multimodal human-computer collaborative interaction system and method. Background Art

[0002] Through multimodal technology, people and machines can interact with each other in a collaborative way. By integrating multiple interaction methods, the communication between people and machines can be made more natural, efficient and intelligent. In shopping malls, there are often intelligent question-and-answer robots that can help customers find stores or specific locations. These intelligent question-and-answer robots are usually equipped with speech recognition and speech synthesis technology, and can communicate with customers through voice communication. Customers can achieve collaborative interaction with intelligent question-and-answer robots through multimodal information such as voice commands and touch screen operations. However, the existing intelligent question-and-answer robot system usually responds according to the preset speaking speed, and cannot be intelligently adjusted for each person's speaking speed. This means that when each person communicates with the intelligent question-and-answer robot, the robot's speaking speed may not conform to the individual's speaking speed habits, resulting in poor communication or discomfort. Therefore, according to each person's speaking speed, the intelligent question-and-answer robot's own speaking speed is adjusted, so that the intelligent question-and-answer robot can better meet the needs of users when responding, which helps to improve the efficiency and fluency of human-machine collaborative interaction, thereby bringing a more personalized communication experience. Summary of the invention

[0003] The purpose of the present invention is to provide a multimodal human-computer collaborative interaction system and method to solve the problems raised in the prior art.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A multimodal human-computer collaborative interaction system and method comprises the following steps:

[0006] Step S100: collecting a surveillance video shot by the intelligent question-answering robot and in which a moving object exists, dividing the surveillance video into a surveillance picture and a surveillance audio, and analyzing the surveillance video to extract from the surveillance video a target moving object that is communicating with the intelligent question-answering robot and a human voice corresponding to the target moving object;

[0007] Step S200: parsing the speech text corresponding to the human voice and analyzing it to obtain the human voice speed corresponding to the human voice; obtaining the response text corresponding to the human voice through the pre-trained large language model, and obtaining each keyword in the response text, and obtaining the key coefficient of each keyword according to the semantic relationship between each keyword and the speech text;

[0008] Step S300: obtaining a feature weight corresponding to each keyword according to a key coefficient corresponding to each keyword, and obtaining a response speech speed corresponding to each keyword by taking the speech speed corresponding to the human voice as an adjustment basis;

[0009] Step S400: According to the response speech rate and the word count of any two adjacent keywords, the response speech rate of the text between the two adjacent keywords is adjusted to obtain the response voice corresponding to the response text.

[0010] Furthermore, step S100 includes the following steps:

[0011] Step S110: obtain the real-time monitoring video starting from the current moment and going forward for S seconds, divide the real-time monitoring video into several frames of monitoring screens, and sort all the monitoring screens from the back to the front in sequence according to the monitoring moments corresponding to the monitoring screens starting from the current moment; construct a neural network model, mark all moving objects in a certain frame of monitoring screen, and the rectangular area box corresponding to the mouth area of ​​each moving object on a certain frame of monitoring screen; obtain the gray value of each pixel in a certain rectangular area box, and aggregate the same gray value to obtain the proportion of each gray value, and then establish a gray histogram corresponding to a certain rectangular area box; according to the gray histogram of a certain rectangular area box in the i-th frame of monitoring screen and the i+1-th frame of monitoring screen, obtain the histogram similarity GS corresponding to the two gray histograms, if the histogram similarity GS>GS 0 , G.S. 0 is the histogram similarity threshold, and the i-th frame of the monitoring screen is marked first. If the histogram similarity GS≤GS 0 , mark the i-th frame of the monitoring screen for the second time, where 1≤i <i+1;

[0012] It should be noted that when a person is talking, the shape of the mouth is constantly changing, resulting in different shapes of the mouth in different frames of monitoring images, resulting in different grayscale values ​​of the mouth area in different frames of monitoring images. Therefore, this scheme uses the histogram similarity corresponding to the two grayscale histograms to characterize the change in the shape of the mouth. If the histogram similarity value is small, it means that the change in the shape of the mouth is not obvious, and it is judged that the person is not talking at this time; if the histogram similarity value is large, it means that the change in the shape of the mouth is more obvious, and it is judged that the person is talking at this time;

[0013] Step S120: Acquire all local regions of each moving object in a certain frame of monitoring picture, where the local regions include eyes, ears, nose and mouth of the face; if the number of local regions of a certain moving object is greater than a quantity threshold, feature mark the certain moving object in the certain frame of monitoring picture;

[0014] Step S130: According to the sorted monitoring screens, if the monitoring screens from the 1st frame to the nth frame are all marked with the first mark and the monitoring screen of the n+1th frame is marked with the second mark, and the monitoring screen of the mth frame is marked with the second mark and the monitoring screens of the m+1th frame to the m+nth frame are all marked with the first mark, and a certain moving object in the monitoring screens of the nth frame to the mth frame is a characteristic mark, then the monitoring video between the monitoring screens of the nth frame to the mth frame and the corresponding monitoring audio are used as the characteristic audio, wherein n and m are the first screen threshold and the second screen threshold, respectively, and 1 <n<m;

[0015] Step S140: Obtain the total number of frames marked with the second mark between the nth frame monitoring screen and the mth frame monitoring screen, and divide it by the total number of frames between the nth frame monitoring screen and the mth frame monitoring screen to obtain the frame ratio R G ; Preprocess the feature audio, and obtain the total time when the amplitude in the feature audio is greater than the amplitude threshold, and divide it by the total time corresponding to the feature audio to obtain the audio ratio R A ; If | R G -R A |≤R 0 , where R 0 is the ratio threshold, the certain moving object is taken as the target moving object, and the characteristic audio is taken as the human voice.

[0016] The moving object in this scheme is the human body, and the preprocessing includes noise removal, signal enhancement, normalization, etc. The main purpose of the preprocessing is to improve the clarity of human voice. In this scheme, if there are normal chats and phone calls between people, although sounds will be made, they are not talking to the robot, and the face of the person is not necessarily pointing to the intelligent question-answering robot during normal chats and phone calls, so this type of human body cannot be used as the target human body. Therefore, this scheme conducts a comprehensive analysis of human voice, human mouth shape and local area. If there is a human voice, but the human body is not in the surveillance video, or in the surveillance video but facing away from the robot, and the human body cannot be judged, then it is judged that the human body is not talking to the robot. The purpose of this step is to find the target human body that is actually talking to the robot by analyzing the mouth shape and local area of ​​the human body.

[0017] Further, step S200 includes the following steps:

[0018] Step S210: Analyze the speech text corresponding to the human voice using a speech recognition algorithm to obtain the total number of words N in the speech text h , and according to the speech time T corresponding to the human voice h , get the human voice speed P corresponding to the human voice h =N h / Th ;

[0019] Step S220: preprocess the speech text and extract relations to obtain all entities and definition relations in the speech text, and obtain the knowledge graph corresponding to the speech text based on the entities and definition relations; obtain each keyword in the response text, obtain the semantic similarity between a certain keyword a and each entity and each definition relation in the knowledge graph, and normalize the semantic similarity. If there is a semantic similarity greater than the semantic similarity threshold, set the quantitative weight to 1, and then obtain the final quantitative weight Q; obtain the key coefficient K corresponding to the certain keyword a a =1-e -Q , where 0≤Q, e is the natural logarithm, 0≤K a <1; and then obtain the key coefficient corresponding to each keyword.

[0020] Knowledge graph is a technology that represents knowledge and information in a graphical way. It consists of entities and definition relationships. In this solution, the large language model is a word vector model. The large language model refers to a type of model that can process natural language, and the word vector model is one of them. The existing Word2Vec can calculate the semantic similarity between two words. The specific technology can be obtained through cosine similarity and normalize the semantic similarity.

[0021] Further, step S300 includes the following steps:

[0022] Step S310: According to the key coefficient corresponding to each keyword, the feature weight corresponding to a keyword a is obtained as follows: , where K a is the key coefficient corresponding to a keyword a, H is the total number of keywords, K h is the key coefficient corresponding to a keyword h;

[0023] Step S320: Get the response speech rate corresponding to a keyword a as P a =(1-K a )*P h / (1+V a ), if P a <P 1 , let P a =P 1 , if P a >P 2 , let P a =P 2 , where K a is the key coefficient corresponding to a keyword a, P h is the human voice speed corresponding to the human voice, P 1 and P 2are the first speech rate threshold and the second speech rate threshold respectively.

[0024] According to the key coefficient corresponding to each keyword, if the key coefficient of a keyword is larger, it means that this keyword covers more effective information. In order to better facilitate human understanding, the corresponding response speed should be slower so that the human body can understand and digest the key information.

[0025] Furthermore, step S400 includes the following steps:

[0026] Step S410: Obtain any two adjacent keywords A and B, and the response speech rate P of keyword A A and the response rate P of keyword B B ; The word count of keyword A and keyword B is recorded as W A and W B , if 1 / L≤W A / W B ≤L, get the response speed of the text between keyword A and keyword B =(P A *W A +P B *W B ) / (W A +W B ), where L is the word count ratio threshold;

[0027] When the number of words in keyword A and keyword B is not much different, a middle value should be taken according to the response speed of keyword A and keyword B. When the number of words in keyword A and keyword B is too different, W should be adjusted to suit the human body. A and W B The maximum value of the response speed is used as the response speed of the text between keyword A and keyword B. In this embodiment, L=1.5. Adjusting the robot pronunciation is essentially adjusting the parameters of the speech synthesis engine. The parameters of the speech synthesis engine include speech speed, pitch, volume, and timbre. In this solution, the speech speed is the object of our study. By adjusting the speech speed of the keywords in the response text or the text between the keywords, the speech synthesis engine can change the reading speed of the entire text by adjusting the speech speed.

[0028] Step S420: If W A / W B <1 / L or W A / W B >L, change W A and W BThe response speech rate corresponding to the maximum value in is taken as the response speech rate of the text between keyword A and keyword B; and then according to the response speech rate of each keyword and the response speech rate of the text between any two adjacent keywords, the response voice corresponding to the response text is obtained.

[0029] A multi-modal human-computer collaborative interaction system, comprising a target moving object judgment module, a key coefficient calculation module, a response speech speed calculation module and a response voice obtaining module;

[0030] Target moving object judgment module: used to collect monitoring videos shot by the intelligent question-answering robot with moving objects in the picture, divide the monitoring videos into monitoring pictures and monitoring audio, analyze the monitoring videos, extract the target moving object that communicates with the intelligent question-answering robot and the human voice corresponding to the target moving object from the monitoring videos;

[0031] Key coefficient calculation module: used to parse the speech text corresponding to the human voice and analyze it to obtain the human voice speed corresponding to the human voice; obtain the response text corresponding to the human voice through the pre-trained large language model, and obtain each keyword in the response text, and obtain the key coefficient of each keyword according to the semantic relationship between each keyword and the speech text;

[0032] Response speed calculation module: used to obtain the feature weight corresponding to each keyword according to the key coefficient corresponding to each keyword, and to obtain the response speed corresponding to each keyword based on the human voice speed corresponding to the human voice;

[0033] Obtaining response voice module: used to adjust the response speed of the text between any two adjacent keywords according to the response speed and the number of keywords, and then obtain the response voice corresponding to the response text.

[0034] Further, the target moving object determination module includes establishing a grayscale histogram unit, a feature marking unit, obtaining a feature audio unit and obtaining a target moving object unit;

[0035] Establishing a grayscale histogram unit: used to obtain the real-time monitoring video starting from the current time and going forward S seconds, and divide the real-time monitoring video into several frames of monitoring screens; marking all moving objects in a certain frame of monitoring screen, and the rectangular area box corresponding to the mouth area of ​​each moving object on a certain frame of monitoring screen, and establishing a grayscale histogram corresponding to a certain rectangular area box; according to the grayscale histogram corresponding to a certain rectangular area box, obtaining the histogram similarity corresponding to the grayscale histogram, and performing the first marking or the second marking on the monitoring screen;

[0036] Feature marking unit: used to obtain a local area of ​​each moving object in a certain frame of monitoring picture; feature marking the moving object in a certain frame of monitoring picture;

[0037] Acquiring characteristic audio units: used to extract characteristic audio from surveillance videos according to the sorted surveillance images;

[0038] Obtaining target moving object unit: used to obtain the picture ratio according to the monitoring picture, obtain the audio ratio according to the characteristic audio, and obtain the target moving object and human voice.

[0039] Further, the module for obtaining the answering voice includes a text answering speech speed calculation unit and an answering voice obtaining unit;

[0040] A text response speech rate calculation unit is used to obtain the response speech rate and the number of words of any two adjacent keywords, and obtain the response speech rate of the text between the any two adjacent keywords;

[0041] Obtaining response voice unit: used for obtaining the response voice corresponding to the response text according to the response speech rate of each keyword and the response speech rate of the text between any two adjacent keywords.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a multimodal human-computer collaborative interaction system and method, including: collecting surveillance videos shot by an intelligent question-and-answer robot with moving objects in the picture, extracting the target moving object and the human voice corresponding to the target moving object; obtaining the human voice speed corresponding to the human voice, obtaining the response text corresponding to the human voice, and obtaining the key coefficient corresponding to each keyword in the response text; obtaining the response speed of each keyword; adjusting the response speed between any two adjacent keywords according to any two adjacent keywords, and then obtaining the response voice corresponding to the response text. The present invention analyzes the surveillance video corresponding to the intelligent question-and-answer robot to obtain the human voice corresponding to the moving object, and adjusts the robot's speech speed to better meet the needs of the user according to the human voice, which helps to improve the efficiency and fluency of human-computer collaborative interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the multi-modal human-computer collaborative interaction method of the present invention;

[0044] Figure 2 It is a structural diagram of the multimodal human-computer collaborative interaction system of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1-Figure 2 The present invention provides a multi-modal human-computer collaborative interaction system and method, and the corresponding multi-modal human-computer collaborative interaction method flowchart is as follows Figure 1 As shown. The following steps are included:

[0047] Step S100: collecting a surveillance video shot by the intelligent question-answering robot and in which a moving object exists, dividing the surveillance video into a surveillance picture and a surveillance audio, and analyzing the surveillance video to extract from the surveillance video a target moving object that is communicating with the intelligent question-answering robot and a human voice corresponding to the target moving object;

[0048] Step S110: obtain the real-time monitoring video starting from the current moment and going forward for S seconds, divide the real-time monitoring video into several frames of monitoring screens, and sort all the monitoring screens from the back to the front in sequence according to the monitoring moments corresponding to the monitoring screens starting from the current moment; construct a neural network model, mark all moving objects in a certain frame of monitoring screen, and the rectangular area box corresponding to the mouth area of ​​each moving object on a certain frame of monitoring screen; obtain the gray value of each pixel in a certain rectangular area box, and aggregate the same gray value to obtain the proportion of each gray value, and then establish a gray histogram corresponding to a certain rectangular area box; according to the gray histogram of a certain rectangular area box in the i-th frame of monitoring screen and the i+1-th frame of monitoring screen, obtain the histogram similarity GS corresponding to the two gray histograms, if the histogram similarity GS>GS 0 , G.S. 0 is the histogram similarity threshold, and the i-th frame of the monitoring screen is marked first. If the histogram similarity GS≤GS 0 , mark the i-th frame of the monitoring screen for the second time, where 1≤i <i+1。

[0049] It should be noted that when a person is speaking, the shape of the mouth is constantly changing, resulting in different mouth shapes in different frames of monitoring images, resulting in different grayscale values ​​of the mouth area in different frames of monitoring images. Therefore, this solution uses the histogram similarity corresponding to the two grayscale histograms to characterize the change in the shape of the mouth. If the histogram similarity value is small, it means that the mouth shape change is not obvious, and it is judged that the person is not speaking at this time; if the histogram similarity value is large, it means that the mouth shape change is more obvious, and it is judged that the person is speaking at this time. The S in the forward S seconds is to limit the speaking time of the human voice. Under normal circumstances, when a human body asks the robot for directions, the time will not be too long. In this embodiment, S=10.

[0050] Step S120: Acquire all local areas of each moving object in a certain frame of monitoring image, where the local areas include eyes, ears, nose and mouth of the face; if the number of local areas of a certain moving object is greater than a quantity threshold, feature mark the certain moving object in the certain frame of monitoring image.

[0051] Since the local areas include the eyes, ears, nose and mouth of the face, a total of 6, people should generally face the robot when speaking. In this solution, the number threshold is 4, which means that when the number of local areas is 5 or 6, it means that the human body is facing the robot, which means that there is an intention to talk to the robot; when the number of local areas is small, it means that the person does not intend to talk to the robot.

[0052] Step S130: According to the sorted monitoring screens, if the monitoring screens from the 1st frame to the nth frame are all marked with the first mark and the monitoring screen of the n+1th frame is marked with the second mark, and the monitoring screen of the mth frame is marked with the second mark and the monitoring screens of the m+1th frame to the m+nth frame are all marked with the first mark, and a certain moving object in the monitoring screens of the nth frame to the mth frame is a characteristic mark, then the monitoring video between the monitoring screens of the nth frame to the mth frame and the corresponding monitoring audio are used as the characteristic audio, wherein n and m are the first screen threshold and the second screen threshold, respectively, and 1 <n<m。

[0053] Step S140: Obtain the total number of frames marked with the second mark between the nth frame monitoring screen and the mth frame monitoring screen, and divide it by the total number of frames between the nth frame monitoring screen and the mth frame monitoring screen to obtain the frame ratio R G ; Preprocess the feature audio, and obtain the total time when the amplitude in the feature audio is greater than the amplitude threshold, and divide it by the total time corresponding to the feature audio to obtain the audio ratio R A ; If | R G -R A |≤R 0 , where R 0is the ratio threshold, the certain moving object is taken as the target moving object, and the characteristic audio is taken as the human voice.

[0054] Preprocessing includes noise removal, signal enhancement, normalization, etc. The main purpose of preprocessing is to improve the clarity of human voice. In this solution, if there are normal chats and phone calls between people, although sounds will be made, they are not talking to the robot, and the face of the person is not necessarily pointing to the intelligent question-answering robot during normal chats and phone calls, so this type of human body cannot be used as the target human body. Therefore, this solution conducts a comprehensive analysis of human voice, human mouth shape and local area. If there is a human voice, but the human body is not in the surveillance video, or in the surveillance video but facing away from the robot, and the human body cannot be judged, then it is judged that the human body is not talking to the robot. The purpose of this step is to find the target human body that is actually talking to the robot by analyzing the human mouth shape and local area.

[0055] Step S200: parsing the speech text corresponding to the human voice and analyzing it to obtain the human voice speed corresponding to the human voice; obtaining the response text corresponding to the human voice through the pre-trained large language model, and obtaining each keyword in the response text, and obtaining the key coefficient of each keyword according to the semantic relationship between each keyword and the speech text;

[0056] Step S210: Analyze the speech text corresponding to the human voice using a speech recognition algorithm to obtain the total number of words N in the speech text h , and according to the speech time T corresponding to the human voice h , get the human voice speed P corresponding to the human voice h =N h / T h .

[0057] Step S220: Preprocess the speech text and perform relationship extraction to obtain all entities and definition relationships in the speech text, and obtain the knowledge graph corresponding to the speech text based on the entities and definition relationships; obtain each keyword in the response text, obtain the semantic similarity between a keyword a and each entity and each definition relationship in the knowledge graph, and normalize the semantic similarity. If there is a semantic similarity greater than the semantic similarity threshold, set the quantitative weight to 1, and then obtain the final quantitative weight Q; obtain the key coefficient K corresponding to a keyword a a =1-e -Q , where 0≤Q, e is the natural logarithm, 0≤K a <1; and then obtain the key coefficient corresponding to each keyword.

[0058] Knowledge graph is a technology that represents knowledge and information in a graphical way. It consists of entities and definition relationships. Entities include people, places, things, concepts, etc. Each entity has a unique identifier and can contain attribute information about the entity, such as name, description, type, etc. Definition relationships represent the relationship between entities. Relationships can be of various types, such as classification relationships, attribute relationships, semantic relationships, etc. Definition relationships can also contain attribute information, such as the type, strength, direction, etc. of the relationship.

[0059] In this solution, the large language model is a word vector model. The large language model refers to a type of model that can process natural language, and the word vector model is one of them. The existing Word2Vec can calculate the semantic similarity between two words. The specific technology can be obtained through cosine similarity, and the semantic similarity is normalized. The specific implementation process is not repeated here. Semantic similarity is a representation of the semantic proximity between two words. It reflects the similarity of the two words in terms of meaning, concept, and object. If the semantic similarity of two words is high, it means that they express similar meanings or describe similar things to a large extent. For example, "how many floors" and "first floor" have a certain degree of semantic similarity because they both represent floor locations and have some common attributes. However, the semantic similarity of "front" and "first floor" is very low because they belong to completely different conceptual categories, one represents direction and the other represents a specific floor.

[0060] Step S300: obtaining a feature weight corresponding to each keyword according to a key coefficient corresponding to each keyword, and obtaining a response speech speed corresponding to each keyword by taking the speech speed corresponding to the human voice as an adjustment basis;

[0061] Step S310: According to the key coefficient corresponding to each keyword, the feature weight corresponding to a keyword a is obtained as follows: , where K a is the key coefficient corresponding to a keyword a, H is the total number of keywords, K h is the key coefficient corresponding to a keyword h.

[0062] Step S320: Get the response speech rate corresponding to a keyword a as P a =(1-K a )*P h / (1+V a ), if P a <P 1 , let P a =P 1 , if P a >P 2 , let P a =P 2 , where K ais the key coefficient corresponding to a keyword a, P h is the human voice speed corresponding to the human voice, P 1 and P 2 are the first speech rate threshold and the second speech rate threshold respectively.

[0063] According to the key coefficient corresponding to each keyword, if the key coefficient of a keyword is larger, it means that this keyword covers more effective information. In order to better facilitate human understanding, the corresponding response speed should be slower so that the human body can understand and digest the key information.

[0064] Step S400: According to the response speech rate and the word count of any two adjacent keywords, the response speech rate of the text between the two adjacent keywords is adjusted to obtain the response voice corresponding to the response text.

[0065] Step S410: Obtain any two adjacent keywords A and B, and the response speech rate P of keyword A A and the response rate P of keyword B B ; The word count of keyword A and keyword B is recorded as W A and W B , if 1 / L≤W A / W B ≤L, get the response speed of the text between keyword A and keyword B =(P A *W A +P B *W B ) / (W A +W B ), where L is the word count ratio threshold.

[0066] When the number of words in keyword A and keyword B is not much different, a middle value should be taken according to the response speed of keyword A and keyword B. When the number of words in keyword A and keyword B is too different, W should be adjusted to suit the human body. A and W B The maximum value of the response speed is used as the response speed of the text between keyword A and keyword B. In this embodiment, L=1.5. Adjusting the robot pronunciation is essentially adjusting the parameters of the speech synthesis engine. The parameters of the speech synthesis engine include speech speed, pitch, volume, and timbre. In this solution, the speech speed is the object of our study. By adjusting the speech speed of the keywords in the response text or the text between the keywords, the speech synthesis engine can change the reading speed of the entire text by adjusting the speech speed.

[0067] Step S420: If W A / W B <1 / L or WA / W B >L, change W A and W B The response speech rate corresponding to the maximum value in is taken as the response speech rate of the text between keyword A and keyword B; and then according to the response speech rate of each keyword and the response speech rate of the text between any two adjacent keywords, the response voice corresponding to the response text is obtained.

[0068] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0069] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Human-computer collaborative interaction method under multimodality, It is characterized in that The following steps are involved: Step S100: collecting a surveillance video shot by the intelligent question-answering robot and in which a moving object exists, dividing the surveillance video into a surveillance picture and a surveillance audio, and analyzing the surveillance video to extract from the surveillance video a target moving object that is communicating with the intelligent question-answering robot and a human voice corresponding to the target moving object; Step S200: parsing the speech text corresponding to the human voice and analyzing it to obtain the human voice speed corresponding to the human voice; obtaining the response text corresponding to the human voice through the pre-trained large language model, and obtaining each keyword in the response text, and obtaining the key coefficient of each keyword according to the semantic relationship between each keyword and the speech text; Step S210: Analyze the speech text corresponding to the human voice using a speech recognition algorithm to obtain the total number of words N in the speech text h , and according to the speech time T corresponding to the human voice h , get the human voice speed P corresponding to the human voice h =N h / T h ; Step S220: preprocess the speech text and extract relations to obtain all entities and definition relations in the speech text, and obtain the knowledge graph corresponding to the speech text based on the entities and definition relations; obtain each keyword in the response text, obtain the semantic similarity between a certain keyword a and each entity and each definition relation in the knowledge graph, and normalize the semantic similarity. If there is a semantic similarity greater than the semantic similarity threshold, set the quantitative weight to 1, and then obtain the final quantitative weight Q; obtain the key coefficient K corresponding to the certain keyword a a =1-e -Q , where 0≤Q, e is the natural logarithm, 0≤K a <1; and then get the key coefficient corresponding to each keyword; Step S300: obtaining a feature weight corresponding to each keyword according to a key coefficient corresponding to each keyword, and obtaining a response speech speed corresponding to each keyword by taking the human voice speech speed corresponding to the human voice as an adjustment basis; Step S310: According to the key coefficient corresponding to each keyword, the feature weight corresponding to a keyword a is obtained as follows: , where K a is the key coefficient corresponding to a keyword a, H is the total number of keywords, K h is the key coefficient corresponding to a keyword h; Step S320: Get the response speech rate corresponding to a keyword a as P a =(1-K a )*P h / (1+V a ), if P a <P 1 , let P a =P 1 , if P a >P 2 , let P a =P 2 , where K a is the key coefficient corresponding to a keyword a, P h is the human voice speed corresponding to the human voice, P 1 and P 2 are the first speech rate threshold and the second speech rate threshold respectively; Step S400: adjusting the response speech rate of the text between any two adjacent keywords according to the response speech rate and the number of the keywords, thereby obtaining a response voice corresponding to the response text; Step S410: Obtain any two adjacent keywords A and B, and the response speech rate P of keyword A A and the response rate P of keyword B B ; The word count of keyword A and keyword B is recorded as W A and W B , if 1 / L≤W A / W B ≤L, get the response speed of the text between keyword A and keyword B =(P A *W A +P B *W B ) / (W A +W B ), where L is the word count ratio threshold; Step S420: If W A / W B <1 / L or W A / W B >L, change W A and W B The response speech rate corresponding to the maximum value in is taken as the response speech rate of the text between keyword A and keyword B; and then according to the response speech rate of each keyword and the response speech rate of the text between any two adjacent keywords, the response voice corresponding to the response text is obtained.

2. The multimodal human-computer collaborative interaction method according to claim 1, It is characterized in that Step S100 includes the following steps: Step S110: obtain the real-time monitoring video starting from the current moment and going forward for S seconds, divide the real-time monitoring video into several frames of monitoring screens, and sort all the monitoring screens from the back to the front in sequence according to the monitoring moments corresponding to the monitoring screens starting from the current moment; construct a neural network model, mark all moving objects in a certain frame of monitoring screen, and the rectangular area box corresponding to the mouth area of ​​each moving object on a certain frame of monitoring screen; obtain the gray value of each pixel in a certain rectangular area box, and aggregate the same gray value to obtain the proportion of each gray value, and then establish a gray histogram corresponding to a certain rectangular area box; according to the gray histogram of a certain rectangular area box in the i-th frame of monitoring screen and the i+1-th frame of monitoring screen, obtain the histogram similarity GS corresponding to the two gray histograms, if the histogram similarity GS>GS 0 , G.S. 0 is the histogram similarity threshold, and the i-th frame of the monitoring screen is marked first. If the histogram similarity GS≤GS 0 , mark the i-th frame of the monitoring screen for the second time, where 1≤i <i+1; Step S120: Acquire all local regions of each moving object in a certain frame of monitoring picture, where the local regions include eyes, ears, nose and mouth of the face; if the number of local regions of a certain moving object is greater than a quantity threshold, feature mark the certain moving object in the certain frame of monitoring picture; Step S130: According to the sorted monitoring screens, if the monitoring screens from the 1st frame to the nth frame are all marked with the first mark and the monitoring screen of the n+1th frame is marked with the second mark, and the monitoring screen of the mth frame is marked with the second mark and the monitoring screens of the m+1th frame to the m+nth frame are all marked with the first mark, and a certain moving object in the monitoring screens of the nth frame to the mth frame is a characteristic mark, then the monitoring video between the monitoring screens of the nth frame to the mth frame and the corresponding monitoring audio are used as the characteristic audio, wherein n and m are the first screen threshold and the second screen threshold, respectively, and 1 <n<m; Step S140: Obtain the total number of frames marked with the second mark between the nth frame monitoring screen and the mth frame monitoring screen, and divide it by the total number of frames between the nth frame monitoring screen and the mth frame monitoring screen to obtain the frame ratio R G ; Preprocess the feature audio, and obtain the total time when the amplitude in the feature audio is greater than the amplitude threshold, and divide it by the total time corresponding to the feature audio to obtain the audio ratio R A ; If | R G -R A |≤R 0 , where R 0 is the ratio threshold, the certain moving object is taken as the target moving object, and the characteristic audio is taken as the human voice.

3. A multimodal human-computer collaborative interaction system, used to execute the multimodal human-computer collaborative interaction method according to any one of claims 1 to 2, It is characterized in that The system includes a target moving object judgment module, a key coefficient calculation module, a response speech speed calculation module and a response voice acquisition module; Target moving object judgment module: used to collect monitoring videos shot by the intelligent question-answering robot with moving objects in the picture, divide the monitoring videos into monitoring pictures and monitoring audio, analyze the monitoring videos, extract the target moving object that communicates with the intelligent question-answering robot and the human voice corresponding to the target moving object from the monitoring videos; Key coefficient calculation module: used to parse the speech text corresponding to the human voice and analyze it to obtain the human voice speed corresponding to the human voice; obtain the response text corresponding to the human voice through the pre-trained large language model, and obtain each keyword in the response text, and obtain the key coefficient of each keyword according to the semantic relationship between each keyword and the speech text; Response speed calculation module: used to obtain the feature weight corresponding to each keyword according to the key coefficient corresponding to each keyword, and to obtain the response speed corresponding to each keyword based on the human voice speed corresponding to the human voice; Obtaining response voice module: used to adjust the response speed of the text between any two adjacent keywords according to the response speed and the number of keywords, and then obtain the response voice corresponding to the response text.

4. The multimodal human-computer collaborative interaction system according to claim 3, It is characterized in that The target moving object determination module includes a grayscale histogram establishment unit, a feature marking unit, a feature audio acquisition unit and a target moving object acquisition unit; Establishing a grayscale histogram unit: for obtaining a real-time monitoring video starting from the current time and S seconds ahead, and dividing the real-time monitoring video into a number of monitoring frames; Mark all moving objects in a certain frame of monitoring screen, and the rectangular area box corresponding to the mouth area of ​​each moving object on the certain frame of monitoring screen, and establish a grayscale histogram corresponding to the rectangular area box; according to the grayscale histogram corresponding to the rectangular area box, obtain the histogram similarity corresponding to the grayscale histogram, and perform the first mark or the second mark on the monitoring screen; Feature marking unit: used to obtain the local area of ​​each moving object in a certain frame of monitoring picture; Mark a moving object in a certain frame of surveillance image with features; Acquiring characteristic audio units: used to extract characteristic audio from surveillance videos according to the sorted surveillance images; Obtaining target moving object unit: used to obtain the picture ratio according to the monitoring picture, obtain the audio ratio according to the characteristic audio, and obtain the target moving object and human voice.

5. The multimodal human-computer collaborative interaction system according to claim 3, It is characterized in that The answering voice obtaining module comprises a text answering speech speed calculating unit and an answering voice obtaining unit; A text response speech rate calculation unit is used to obtain the response speech rate and the number of words of any two adjacent keywords, and obtain the response speech rate of the text between the any two adjacent keywords; Obtaining response voice unit: used for obtaining the response voice corresponding to the response text according to the response speech rate of each keyword and the response speech rate of the text between any two adjacent keywords.

Images