Speaking behavior detection method, robot control equipment and readable storage medium

Abstract

The invention provides a speaking behavior detection method, robot control equipment and a readable storage medium, and relates to the technical field of computer vision. According to the invention, the actual lip distance of the target person in the target face image at the current moment and the historical lip distances of the target person in the historical face images at multiple continuous historical moments before the current moment are combined to determine that the actual speaking trend state of the target person at the current moment is the suspected speaking state; in the embodiment of the invention, state statistics is carried out on the actual speaking trend state and the historical speaking trend states of the target person at a plurality of continuous historical moments before the current moment, and when the real-time state distribution proportion of the suspected speaking state exceeds a preset state proportion threshold, the target person is judged to speak at the current moment. And otherwise, judging that the target person does not speak at the current moment, thereby effectively improving the speaking behavior detection accuracy through the combination of lip distance change detection of continuous multi-frame face images and a voting mechanism, and improving the speaking behavior missing / false detection problem.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a speaking behavior detection method, a robot control device, and a readable storage medium. Background Art

[0002] With the continuous development of science and technology, computer vision technology is increasingly used in video surveillance, social security and other fields. Among them, the human-computer interaction design of intelligent robots is an important research direction of computer vision technology today. In the process of human-computer interaction of intelligent robots, it is usually necessary to detect and judge the user's speaking behavior through computer vision technology, so as to promptly confirm whether the user has finished speaking, and intercept the voice text just spoken by the user to generate answers using a large language model for dialogue, or promptly confirm whether the user is waking up the intelligent robot. Therefore, in the process of human-computer interaction of intelligent robots, how to effectively improve the detection accuracy of user speaking behavior and avoid missed detection or false detection is an important problem that needs to be solved in computer vision technology today. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a speaking behavior detection method, a robot control device and a readable storage medium, which can effectively and stably characterize the user's speaking behavior change process over a period of time based on the lip distance changes of multiple consecutive frames of facial images, so as to determine the changes in the user's speaking trend state during the human-computer interaction process, and then ensure through a voting mechanism that the final output user speaking behavior detection result meets the high precision requirements, thereby effectively improving the problem of missed or false detection of user speaking behavior, and basically eliminating the negative impact of the disturbance of key points of the facial image caused by external factors (for example, changes in the light of the facial image shooting environment, adjustment of the user's facial posture, etc.) on the speaking behavior detection operation.

[0004] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0005] In a first aspect, the present application provides a method for detecting speech behavior, the method comprising:

[0006] Obtaining a target face image of a target person at a current moment, and calculating an actual lip distance of the target person in the target face image;

[0007] Determine whether the actual speaking trend state of the target person at the current moment is a suspected speaking state or not according to the actual lip distance and the historical lip distance of the target person in the historical face images of a plurality of consecutive historical moments before the current moment;

[0008] In the case where the actual speaking trend state is a suspected speaking state, status statistics are performed on the actual speaking trend state and the historical speaking trend states of the target person at a plurality of consecutive historical moments before the current moment, to obtain a real-time status distribution ratio of the suspected speaking state at the current moment;

[0009] Detecting whether the real-time status distribution ratio exceeds a preset status ratio threshold;

[0010] If it is detected that the real-time state distribution ratio exceeds the preset state ratio threshold, it is determined that the target person is speaking at the current moment; otherwise, it is determined that the target person is not speaking at the current moment.

[0011] In an optional implementation, the step of calculating the actual lip distance of the target person in the target face image includes:

[0012] Calling a pre-trained facial key point detection model to perform facial key point detection on the target facial image to obtain lip key point information in the target facial image;

[0013] Calculate the upper and lower inner lip distances corresponding to each of the multiple groups of inner lip key points in the target face image according to the lip key point information, wherein each group of inner lip key points consists of an upper inner lip key point and a lower inner lip key point;

[0014] The upper and lower inner lip distances corresponding to the multiple groups of inner lip key points are averaged to obtain the actual lip distance of the target person in the target face image.

[0015] In an optional implementation, the step of determining whether the actual speaking trend state of the target person at the current moment is a suspected speaking state based on the actual lip distance and the historical lip distance of the target person in historical facial images at a plurality of consecutive historical moments before the current moment comprises:

[0016] Calculate the effective lip distance change amplitude value of the target person at the current moment according to the actual lip distance and the historical lip distance of the target person in the historical face images of a plurality of consecutive historical moments before the current moment;

[0017] Detecting whether the effective lip distance change amplitude value exceeds a preset change amplitude threshold;

[0018] If it is detected that the effective lip distance change amplitude value exceeds the preset change amplitude threshold, it is determined that the actual speaking trend state of the target person at the current moment is a suspected speaking state; otherwise, it is determined that the actual speaking trend state of the target person at the current moment is a non-speaking state.

[0019] In an optional implementation, the step of calculating the effective lip distance change amplitude value of the target person at the current moment based on the actual lip distance and the historical lip distance of the target person in historical facial images of a plurality of consecutive historical moments before the current moment comprises:

[0020] Calculating the absolute value of the lip distance difference between the actual lip distance and the historical lip distances of each of the multiple historical moments;

[0021] The maximum lip distance difference absolute value is selected from the lip distance difference absolute values ​​corresponding to the multiple historical moments, and the maximum lip distance difference absolute value is used as the effective lip distance change amplitude value of the target person at the current moment.

[0022] In an optional implementation, the step of calculating the effective lip distance change amplitude value of the target person at the current moment based on the actual lip distance and the historical lip distance of the target person in historical facial images of a plurality of consecutive historical moments before the current moment comprises:

[0023] Calling a pre-trained facial key point detection model to determine the distribution ratio of the first image matching the side face posture of the target person in the target face image and the historical face images of the multiple historical moments, and the distribution ratio of the second image matching the lip area occlusion phenomenon;

[0024] Detecting whether the first image distribution ratio exceeds a first preset image ratio threshold, and detecting whether the second image distribution ratio exceeds a second preset image ratio threshold;

[0025] When it is detected that the first image distribution ratio exceeds the first preset image ratio threshold, and / or when it is detected that the second image distribution ratio exceeds the second preset image ratio threshold, directly setting the effective lip distance change amplitude value of the target person at the current moment to zero;

[0026] When it is detected that the first image distribution ratio does not exceed the first preset image ratio threshold, and when it is detected that the second image distribution ratio does not exceed the second preset image ratio threshold, the absolute value of the lip distance difference between the actual lip distance and the historical lip distance at each of the multiple historical moments is calculated, and the maximum absolute value of the lip distance difference is selected as the effective lip distance change amplitude value of the target person at the current moment.

[0027] In an optional implementation, before executing the step of collecting status statistics of the actual speaking trend status and the historical speaking trend status of the target person at a plurality of consecutive historical moments before the current moment, the method further includes:

[0028] In the case where the actual speaking trend state is a suspected speaking state, detecting whether there is unreasonable movement of the target person in historical facial images of a plurality of consecutive historical moments before the current moment according to the target facial image;

[0029] When it is detected that there is no unreasonable movement phenomenon in the historical facial images of the target person at multiple consecutive historical moments before the current moment, continue to execute the step of performing status statistics of the actual speaking trend state and the historical speaking trend state of the target person at multiple consecutive historical moments before the current moment.

[0030] In an optional implementation, the step of detecting whether there is unreasonable movement in historical facial images of the target person at a plurality of consecutive historical moments before the current moment based on the target facial image comprises:

[0031] According to face frame position information and face frame size information of an actual face frame of the target face image, construct a legal face frame moving area that covers the actual face frame and has a corresponding area size larger than the face frame size information, and according to an actual area value of the actual face frame, construct a legal face frame area interval that includes the actual area value;

[0032] For each historical face image among the multiple historical moments, detecting whether a historical face frame of the historical face image at least partially exceeds the legal face frame moving area, and detecting whether a face frame area value of the historical face frame of the historical face image exceeds the legal face frame area interval;

[0033] When it is detected that a historical face frame of any one of the historical face images at the multiple historical moments at least partially exceeds the legal face frame movement area, and / or when it is detected that a face frame area value of any one of the historical face images at the multiple historical moments exceeds the legal face frame area interval, it is determined that the target person has unreasonable movement in the historical face images at the multiple historical moments;

[0034] When it is detected that the historical face frames of each of the historical face images at the multiple historical moments do not exceed the legal face frame movement area, and the corresponding face frame area values ​​do not exceed the legal face frame area range, it is determined that there is no unreasonable movement of the target person in the historical face images at the multiple historical moments.

[0035] In an optional embodiment, the method further comprises:

[0036] Obtain a labeled training sample set for a facial key point detection model, wherein the labeled training sample set consists of a first facial sample image set labeled with all facial key points and a second facial sample image set labeled with only lip key points;

[0037] The first face sample image set and the second face sample image set are alternately input into the face key point detection model, and the face key point detection model is trained according to the target loss weight vector matched with the input face sample image set, with the purpose of minimizing the target loss function matched with the input face sample image set, so as to obtain the face key point detection model that meets the expected performance state; wherein, the first loss weight vector matched with the first face sample image set and the second loss weight vector matched with the second face sample image set maintain the same vector dimension, the loss weight value of the inner lip key point in the first loss weight vector is greater than the loss weight values ​​of other face key points other than the inner lip key points, the loss weight values ​​of other face key points other than the lip key points in the second loss weight vector are zero, and the loss weight value of the inner lip key point in the second loss weight vector is greater than the loss weight values ​​of other lip key points other than the inner lip key points.

[0038] In a second aspect, the present application provides a robot control device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the speaking behavior detection method described in any one of the aforementioned embodiments.

[0039] In a third aspect, the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a computer device, the method for detecting speaking behavior described in any one of the aforementioned embodiments is implemented.

[0040] In this case, the beneficial effects of the embodiments of the present application may include the following:

[0041] The present application combines the actual lip distance of the target person in the target facial image at the current moment and the historical lip distance of the target person in the historical facial images of multiple consecutive historical moments before the current moment. When it is determined that the actual speaking trend state of the target person at the current moment is a suspected speaking state, the actual speaking trend state and the historical speaking trend state of the target person at multiple consecutive historical moments before the current moment are statistically analyzed, and when the real-time state distribution ratio of the suspected speaking state exceeds a preset state ratio threshold, it is determined that the target person is speaking at the current moment, otherwise it is determined that the target person is not speaking at the current moment, thereby making it possible to use the lip distance changes of multiple consecutive frames of facial images to effectively and stably characterize the user's speaking behavior change process over a period of time, facilitate determining the user's speaking trend state changes during the human-computer interaction process, and then ensure that the final output user speaking behavior detection result meets the high precision requirements through a voting mechanism, so as to improve the problem of missed or false detection of user speaking behavior, and at the same time basically eliminate the negative impact of facial image key point disturbance caused by external factors on the speaking behavior detection operation.

[0042] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the composition of a robot control device provided in an embodiment of the present application;

[0045] Figure 2 One of the flow charts of the speaking behavior detection method provided in the embodiment of the present application;

[0046] Figure 3 for Figure 2 A schematic flow chart of the sub-steps included in step S210;

[0047] Figure 4 A schematic diagram of the distribution of key points of a face provided in an embodiment of the present application;

[0048] Figure 5 for Figure 2 A schematic flow chart of the sub-steps included in step S220;

[0049] Figure 6 for Figure 5 One of the schematic diagrams of the step execution flow of sub-step S221 in FIG.

[0050] Figure 7 for Figure 5 The second schematic diagram of the step execution flow of sub-step S221 in FIG.

[0051] Figure 8 The second flowchart of the speaking behavior detection method provided in the embodiment of the present application;

[0052] Fig. 9 for Figure 8 A flowchart of the sub-steps included in step S270;

[0053] Fig.10 The third flowchart of the speaking behavior detection method provided in the embodiment of the present application.

[0054] Icons: 10 - robot control device; 11 - memory; 12 - processor; 13 - communication unit. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0058] In the description of the present application, it should be understood that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is conventionally placed when in use, or are the orientations or positional relationships conventionally understood by those skilled in the art. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0059] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0060] In addition, in the description of the present application, it is understood that the relational terms such as the term "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood by specific circumstances.

[0061] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0062] Please refer to Figure 1 , Figure 1: is a schematic diagram of the composition of the robot control device 10 provided in the embodiment of the present application. In the embodiment of the present application, the robot control device 10 can be connected to the intelligent robot for communication, and obtain the face image taken by the intelligent robot in real time for the interacting person, so as to effectively and stably characterize the change process of the interactive person's speech behavior over a period of time through the change of the lip distance of multiple consecutive frames of face images, determine the change of the speech trend state of the interactive person in the human-computer interaction process, and then perform high-precision detection of the actual speech behavior of the interactive person in the human-computer interaction process through a voting mechanism based on the determined change of the speech trend state, to ensure that the final output speech behavior detection result can meet the high-precision requirements, and effectively reflect the real speech status of the interactive person, so as to effectively improve the problem of missed or false detection of the user's speech behavior, and basically eliminate the negative impact of the key point disturbance of the face image on the speech behavior detection operation (for example, the phenomenon of large lip distance calculation error due to the key point disturbance of the lip in a single frame of face image). Among them, the robot control device 10 can be a computer device independent of the intelligent robot, and the computer device can be but is not limited to a personal computer, a cloud server, a laptop computer, a tablet computer, etc.; the robot control device 10 can also be a physical hardware device integrated with the intelligent robot.

[0063] In the embodiment of the present application, the robot control device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, the memory 11, the processor 12, and the communication unit 13 may be electrically connected to each other via one or more communication buses or signal lines.

[0064] In the embodiment of the present application, the memory 11 may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 11 is used to store a computer program, and the processor 12 may execute the computer program accordingly after receiving an execution instruction.

[0065] In addition, the memory 11 can also be used to store a pre-trained facial key point detection model, and the facial key point detection model can be used to perform facial key point detection on any facial image of the interacting person to determine the facial key point information in the corresponding facial image (including pixel point position information of a preset number (for example, 68, 98 or 106) of facial key points in the corresponding facial image), so as to filter out lip key point information related to the lips from the facial key point information (including pixel point position information of all lip key points in the corresponding facial image) to calculate the lip distance and obtain the effective lip distance size of the interacting person in the corresponding facial image.

[0066] In the embodiment of the present application, the processor 12 may be an integrated circuit chip with signal processing capability. The processor 12 may be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., which may implement or execute the disclosed methods, steps, and logic block diagrams in the embodiment of the present application.

[0067] In the embodiment of the present application, the communication unit 13 is used to establish a communication connection between the robot control device 10 and other electronic devices through a network, and to send and receive data through the network, wherein the network includes a wired communication network and a wireless communication network. For example, the robot control device 10 can obtain a face image taken by the intelligent robot in real time for the interacting person through the communication unit 13, and feed back the speech behavior detection result obtained for the face image to the intelligent robot, so as to drive the intelligent robot to respond to the interacting person in a timely manner.

[0068] In an embodiment of the present application, the robot control device 10 may pre-store a specific computer program related to the user speaking behavior detection function in the memory 11, and by driving the processor 12 to execute the specific computer program stored in the memory 11, based on the combination of lip distance change detection and voting mechanism of continuous multi-frame facial images, basically eliminate the negative impact of the key point disturbance of the facial image on the speaking behavior detection operation, and improve the problem of missing / false detection of speaking behavior, thereby improving the accuracy of speaking behavior detection, ensuring that the final output speaking behavior detection result can meet the high accuracy requirements and effectively reflect the actual speaking status of the interacting personnel.

[0069] Understandably, Figure 1 The block diagram shown is only a schematic diagram of a composition of the robot control device 10. The robot control device 10 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0070] In the present application, in order to ensure that the robot control device 10 can eliminate the negative impact of the key point disturbance of the face image on the speech behavior detection operation based on the combination of lip distance change detection and voting mechanism of continuous multi-frame face images, and improve the speech behavior leakage / false detection problem, so as to improve the accuracy of speech behavior detection, so that the corresponding speech behavior detection result can reflect the actual speaking status of the interacting person, the embodiment of the present application provides a speech behavior detection method to achieve the above-mentioned purpose. The speech behavior detection method provided by the present application is described in detail below.

[0071] Please refer to Figure 2 , Figure 2 is one of the flow charts of the speech behavior detection method provided in the embodiment of the present application. In the embodiment of the present application, Figure 2 The speaking behavior detection method shown may include steps S210 to S260.

[0072] Step S210, obtaining a target facial image of the target person at the current moment, and calculating the actual lip distance of the target person in the target facial image.

[0073] In this embodiment, the target person is the interactive person of the intelligent robot in the human-computer interaction process; when the robot control device 10 obtains the facial image of the target person taken in real time by the intelligent robot at any time, the moment can be taken as the current moment, and the facial image taken by the intelligent robot at the current moment is taken as the target facial image, and then the pre-trained facial key point detection model is called to perform facial key point detection on the target facial image, so as to calculate the effective lip distance size (i.e., the actual lip distance) of the target person in the target facial image based on the pixel position information of each facial key point in the detected target facial image.

[0074] Optionally, see Figure 3 , Figure 3 yes Figure 2 Flow chart of the sub-steps included in step S210. In an embodiment of the present application, the step of "calculating the actual lip distance of the target person in the target face image" in step S210 may include sub-steps S211 to S213 to calculate the effective lip distance of the target person in the target face image as accurately as possible.

[0075] Sub-step S211, calling a pre-trained facial key point detection model to perform facial key point detection on the target facial image to obtain lip key point information in the target facial image.

[0076] In one implementation of this embodiment, the facial key point detection model may use at least 98 facial key points (such as Figure 4 The face sample image set (0 to 97 face key points in the face key points) is trained to ensure that the face key point detection model can extract 98 face key points that can represent the global relevance of the face from any face image when performing face key point detection on the face image, and then select 20 lip key points related to the lip area from the extracted 98 face key points (such as Figure 4 The facial key points 76-95 in the rectangular frame are obtained to obtain the lip key point information in the facial image (including the pixel position information of each of the 20 lip key points in the corresponding facial image).

[0077] Sub-step S212, calculating the upper and lower inner lip distances corresponding to each of the multiple groups of inner lip key points in the target face image according to the lip key point information, wherein each group of inner lip key points consists of an upper inner lip key point and a lower inner lip key point.

[0078] In this embodiment, because the 20 lip key points related to the lip area are directly used to calculate the lip distance, it is easy to cause the detected lip distance change between two adjacent frames of face images to be inconsistent with the actual situation (for example, the interacting person does not have lip movement, but the calculated lip distance change value is regarded as the lip movement phenomenon). Therefore, the 6 inner lip key points (such as the lip distance of the lip key points) with the largest feature change when the lip movement phenomenon occurs can be used. Figure 4 The lip key points 89-91 and 93-95 in the target face image are used to calculate the lip distance to ensure that the final calculated lip distance can fit the actual lip movement of the interacting person as much as possible. Therefore, after obtaining the lip key point information in the target face image, the robot control device 10 can filter out 3 groups of inner lip key points from the lip key point information (where each group of inner lip key points consists of an upper inner lip key point (for example, Figure 4 any of the lip key points 89-91 in the ) and a lower inner lip key point (for example, Figure 4 , wherein lip key point 89 and lip key point 95 constitute a group of inner lip key points, lip key point 90 and lip key point 94 constitute a group of inner lip key points, and lip key point 91 and lip key point 95 constitute a group of inner lip key points), and then for each group of inner lip key points, the key point distance is calculated according to the pixel position information of the upper inner lip key points and the lower inner lip key points in the group of inner lip key points to obtain the upper and lower inner lip distances of the group of inner lip key points.

[0079] Sub-step S213, performing a mean operation on the upper and lower inner lip distances corresponding to the multiple groups of inner lip key points to obtain the actual lip distance of the target person in the target face image.

[0080] Therefore, the present application can calculate the effective lip distance of the target person in the target face image as accurately as possible by executing the above sub-steps S211 to S213.

[0081] Step S220, determining whether the actual speaking trend state of the target person at the current moment is a suspected speaking state based on the actual lip distance and the historical lip distance in the historical face images of the target person at multiple consecutive historical moments before the current moment.

[0082] In this embodiment, it is taken into consideration that when the interacting person is not speaking, there is basically no change in the lip distance between different facial images. Only when the interacting person is actually speaking, there will be a period of time when there is an obvious change in the lip distance between frames. In other words, the change in the lip distance between two adjacent facial images or any two designated facial images is difficult to reflect the actual speaking trend of the interacting person during the change of his speaking behavior. If the lip distance change between two facial images is directly used to identify the speaking behavior, it is very easy to miss or misdetect the speaking behavior.

[0083] Therefore, after the robot control device 10 calculates the actual lip distance of the target person at the current moment, it will select historical lip distances of multiple historical moments that are continuously distributed with the current moment from the historical lip distances in the historical facial images of the target person at many historical moments before the current moment, according to a first preset number (which is a positive integer greater than 1), so as to effectively and stably characterize the changing process of the interacting person's speaking behavior over a period of time through the lip distance changes between the actual lip distance and the first preset number of historical lip distances, and ensure that the actual speaking trend state of the target person at the current moment can be matched with the actual speaking status of the target person as much as possible, so as to improve the problem of missed or false detection of speaking behavior.

[0084] In one implementation of the present embodiment, the robot control device 10 may construct a lip distance sequence for the target person separately at the memory 11, and record the effective lip distance sizes of the target person at different times within a period of time in chronological order through the lip distance sequence. Therefore, the robot control device 10 may write the actual lip distance of the target person at the current moment into the end of the lip distance sequence, and delete the historical lip distance recorded at the head of the lip distance sequence. At this time, all the historical lip distances recorded in the lip distance sequence except the actual lip distance are the historical lip distances at multiple historical moments continuously distributed with the current moment, wherein the sequence length of the lip distance sequence is obtained by adding one to the first preset number.

[0085] In this embodiment, the speaking tendency state of any interacting person at any time may be a suspected speaking state or a temporarily unspeaking state. The suspected speaking state is used to characterize that the corresponding interacting person may be speaking at the corresponding time, and the temporarily unspeaking state is used to characterize that the corresponding interacting person is not speaking at the corresponding time.

[0086] In this case, when the robot control device 10 determines that the actual speaking trend state of the target person at the current moment is not a suspected speaking state (that is, the actual speaking trend state of the target person at the current moment is a non-speaking state), it can be determined that the target person is regarded as not speaking at the current moment, and the robot control device 10 will jump back to step S210 to continue execution, so as to perform speaking behavior detection on the face image of the target person at the next moment after the current moment (that is, the new current moment); and when the robot control device 10 determines that the actual speaking trend state of the target person at the current moment is a suspected speaking state, the robot control device 10 will correspondingly execute step S230, so as to further confirm whether the target person is actually speaking at the current moment.

[0087] Optionally, see Figure 5 , Figure 5 yes Figure 2 In the embodiment of the present application, the step S220 may include sub-steps S221 to S224, so as to effectively determine the actual speaking trend of the interacting person at the current moment during the change of speaking behavior based on the lip distance change detection of multiple consecutive frames of face images.

[0088] Sub-step S221, calculates the effective lip distance change amplitude value of the target person at the current moment based on the actual lip distance and the historical lip distance in the historical face images of the target person at multiple consecutive historical moments before the current moment.

[0089] In this embodiment, the effective lip distance change amplitude value is used to characterize the target person's speaking behavior change trend at the current moment within a first time period, wherein the upper time limit boundary of the first time period is the current moment, and the first time period is composed of the current moment and a first preset number of historical moments continuously distributed with the current moment.

[0090] Optionally, see Figure 6 , Figure 6 yes Figure 5 In one implementation of the present embodiment, the sub-step S221 may include sub-steps S2211 to S2212 to simplify the process of measuring the change trend of speech behavior at any time and improve the efficiency of measuring the change trend of speech behavior.

[0091] Sub-step S2211, calculating the absolute value of the lip distance difference between the actual lip distance and the historical lip distances at multiple historical moments.

[0092] In this embodiment, for each of the first preset number of historical lip distances, the absolute value of the lip distance difference between the actual lip distance and the historical lip distance can be obtained by calculating the absolute value of the lip distance difference between the actual lip distance and the historical lip distance.

[0093] Sub-step S2212, selecting the maximum lip distance difference absolute value from the lip distance difference absolute values ​​corresponding to multiple historical moments, and using the maximum lip distance difference absolute value as the effective lip distance change amplitude value of the target person at the current moment.

[0094] Therefore, the present application can simplify the process of measuring the trend of speech behavior changes at any time and improve the efficiency of measuring the trend of speech behavior changes by executing the above sub-steps S2211 to S2212.

[0095] Optionally, see Figure 7 , Figure 7 yes Figure 5 The second step execution flow diagram of sub-step S221 in the embodiment. In another implementation of the present embodiment, due to the interference of the external dynamic environment, the interacting person usually has lip occlusion or side face posture phenomenon within a certain time period, resulting in that the lip distance sizes at different times measured by the face key point detection model within the time period contain some data that cannot represent the actual lip movement of the interacting person. Therefore, when measuring the trend of changes in speaking behavior at any time, it is necessary to perform anti-interference processing on the lip occlusion phenomenon and side face posture phenomenon to avoid the false detection problem of non-user interaction caused by the external dynamic environment as much as possible. At this time, the sub-step S221 may include sub-steps S2213 to S2216.

[0096] Sub-step S2213, calls a pre-trained facial key point detection model to determine the first image distribution ratio that matches the side face posture of the target person in the target face image and the historical face images of multiple historical moments, and the second image distribution ratio that matches the lip area occlusion phenomenon.

[0097] In this embodiment, each time the robot control device 10 calls the facial key point detection model to perform facial key point detection on any facial image, it will perform facial regression prediction based on the detected facial key point information to determine whether the facial posture in the corresponding facial image is a side face posture. At the same time, it will also determine the actual lip area in the corresponding facial image based on the detected lip key point information, and then perform occlusion analysis on the image content in the actual lip area to determine whether there is lip area occlusion in the corresponding facial image. Among them, when the facial posture in the corresponding facial image satisfies the posture angle condition "the heading angle is less than 30°, the pitch angle is less than 20°, and the sum of the heading angle, the pitch angle and the roll angle is less than 60°", the corresponding facial posture is determined to be a side face posture, otherwise the corresponding facial posture is determined to be a frontal face posture; the regional center pixel position of the actual lip area in the facial image can be obtained by performing mean calculation on the pixel position information of each of the lip key points in the facial image, and then according to the preset size ratio of the lip area to the facial area (for example, the ratio of the lip area width to the facial area width is 0.7, and the ratio of the lip area height to the facial area height is 0.3), the actual lip area size in the facial image is determined based on the actual facial area size in the facial image, so as to circle the actual lip area range in the facial image.

[0098] Therefore, when the robot control device 10 determines the actual lip distance of the target person at the current moment, and the historical lip distance of the target person at a first preset number of historical moments that are continuously distributed with the current moment, facial posture statistics and lip area occlusion frequency statistics are performed on the person images corresponding to the current moment and the preset number of historical moments, and a first image distribution ratio of the number of facial images with facial postures relative to the total number of facial images (obtained by adding one to the first preset number) and a second image distribution ratio of the number of facial images with lip area occlusion relative to the total number of facial images are obtained, so as to judge the specific influence of the lip occlusion phenomenon and the side face posture phenomenon in the first time period through the two image distribution ratios, wherein the larger the corresponding image distribution ratio, the greater the influence of the lip occlusion phenomenon or the side face posture phenomenon.

[0099] Optionally, the robot control device 10 can construct a facial posture sequence and a lip concealment sequence for the target person at the memory 11, and record in chronological order whether the facial posture of the target person at different moments in a period of time is a side face state through the facial posture sequence, and record in chronological order whether the lip concealment phenomenon of the target person at different moments in a period of time through the lip concealment sequence, wherein the sequence lengths of the facial posture sequence, the lip concealment sequence and the lip distance sequence are consistent. Therefore, the robot control device 10 can write the actual facial posture of the target person at the current moment into the sequence tail of the facial posture sequence, and delete the historical facial posture recorded at the sequence head of the facial posture sequence, at which time the entire facial posture sequence can be directly used to calculate the first image distribution ratio; the robot control device 10 can write the actual lip concealment status of the target person at the current moment into the sequence tail of the lip concealment sequence, and delete the historical lip concealment status recorded at the sequence head of the lip concealment sequence, at which time the entire lip concealment sequence can be directly used to calculate the second image distribution ratio.

[0100] Sub-step S2214, detecting whether the first image distribution ratio exceeds a first preset image ratio threshold, and detecting whether the second image distribution ratio exceeds a second preset image ratio threshold.

[0101] In this embodiment, the first preset image proportion threshold may be consistent with the second preset image proportion threshold, for example, 70%; the first preset image proportion threshold may also be different from the second preset image proportion threshold, for example, the first preset image proportion threshold is 75%, and the second preset image proportion threshold is 65%.

[0102] Sub-step S2215, when it is detected that the first image distribution ratio exceeds the first preset image ratio threshold, and / or when it is detected that the second image distribution ratio exceeds the second preset image ratio threshold, directly set the effective lip distance change amplitude value of the target person at the current moment to zero.

[0103] In this embodiment, when it is detected that the distribution ratio of the first image exceeds the first preset image ratio threshold, it indicates that the side face posture phenomenon caused by the external dynamic environment has seriously affected the lip distance reliability of the target person in the first time period. In order to improve the false detection problem of non-user interaction caused by the external dynamic environment, it is necessary to directly set the effective lip distance change amplitude value of the target person at the current moment to zero; when it is detected that the distribution ratio of the second image exceeds the second preset image ratio threshold, it indicates that the lip occlusion phenomenon caused by the external dynamic environment has seriously affected the lip distance reliability of the target person in the first time period. In order to improve the false detection problem of non-user interaction caused by the external dynamic environment, it is necessary to directly set the effective lip distance change amplitude value of the target person at the current moment to zero.

[0104] Sub-step S2216, when it is detected that the first image distribution ratio does not exceed the first preset image ratio threshold, and when it is detected that the second image distribution ratio does not exceed the second preset image ratio threshold, calculate the absolute value of the lip distance difference between the actual lip distance and the historical lip distance of each of multiple historical moments, and select the maximum absolute value of the lip distance difference as the effective lip distance change amplitude value of the target person at the current moment.

[0105] In this embodiment, when it is detected that the first image distribution ratio does not exceed the first preset image ratio threshold, and the second image distribution ratio does not exceed the second preset image ratio threshold, it indicates that the side face posture phenomenon and lip occlusion phenomenon brought about by the external dynamic environment have almost no effect on the lip distance reliability of the target person in the first time period. At this time, the same processing method as the above-mentioned sub-steps S2211 to SS2212 can be adopted to calculate the effective lip distance change amplitude value of the target person at the current moment.

[0106] Therefore, the present application can execute the above-mentioned sub-steps S2213 to S2216 to perform anti-interference processing on lip occlusion and side face posture phenomena when measuring the changing trend of speaking behavior at any time, so as to avoid the false detection problem of non-user interaction caused by the external dynamic environment as much as possible.

[0107] Sub-step S222, detecting whether the effective lip distance change amplitude value exceeds a preset change amplitude threshold.

[0108] In this embodiment, when it is detected that the effective lip distance change amplitude value of the target person at the current moment exceeds the preset change amplitude threshold, it indicates that the target person may still be speaking at the current moment, and the robot control device 10 will execute sub-step S223 accordingly; when it is detected that the effective lip distance change amplitude value of the target person at the current moment does not exceed the preset change amplitude threshold, it indicates that the target person can be regarded as not speaking at the current moment, and the robot control device 10 will execute sub-step S224 accordingly.

[0109] Sub-step S223, determining whether the actual speaking trend state of the target person at the current moment is a suspected speaking state.

[0110] Sub-step S224, determining whether the actual speaking trend state of the target person at the current moment is a state of not speaking yet.

[0111] Therefore, the present application can effectively determine the actual speaking trend of the interacting person at the current moment during the change of speaking behavior based on the lip distance change detection of multiple consecutive frames of facial images by executing the above sub-steps S221 to S224.

[0112] Step S230 , performing status statistics on the actual speaking trend state and the historical speaking trend state of the target person at a plurality of consecutive historical moments before the current moment, and obtaining a real-time status distribution ratio of the suspected speaking state at the current moment.

[0113] In this embodiment, when the robot control device 10 determines that the actual speaking trend state of the target person at the current moment is a suspected speaking state, it will select the historical speaking trend states of multiple historical moments that are continuously distributed with the current moment from the stored historical speaking trend states of the target person at many historical moments before the current moment according to a second preset number (which is a positive integer greater than 1), so as to use a voting mechanism to perform suspected speaking state statistics on the actual speaking trend state and the second preset number of historical speaking trend states, and determine the real-time state distribution ratio of the number of moments when the suspected speaking state occurs relative to the total number of moments (obtained by adding one to the second preset number). Among them, the first preset number can be consistent with the second preset number, or it can be different from the second preset number.

[0114] Optionally, in one implementation of the present embodiment, the robot control device 10 may construct a separate speaking trend state queue for the target person at the memory 11, and record the specific speaking trend states of the target person at different times within a period of time in chronological order through the speaking trend state queue. Therefore, the robot control device 10 can write the actual speaking trend state of the target person at the current moment into the speaking trend state queue, and delete the historical speaking trend state written first in the speaking trend state queue. At this time, the speaking trend state queue can be directly used to calculate the real-time state distribution ratio of the suspected speaking state at the current moment, wherein the queue length of the speaking trend state queue is obtained by adding one to the second preset number.

[0115] Step S240, detecting whether the real-time status distribution ratio exceeds a preset status ratio threshold.

[0116] In this embodiment, when the real-time status distribution ratio of the target person regarding the suspected speaking state at the current moment exceeds the preset status ratio threshold (for example, 30%), it indicates that the target person is continuously in the speaking state within a second time period, and the robot control device 10 will execute step S250 accordingly; when the real-time status distribution ratio of the target person regarding the suspected speaking state at the current moment does not exceed the preset status ratio threshold, it indicates that the target person does not show a continuous speaking state within the second time period, and the robot control device 10 will execute step S260 accordingly, wherein the upper limit of the second time period is the current moment, and the second time period is composed of the current moment and a second preset number of historical moments that are continuously distributed with the current moment.

[0117] Step S250, determining whether the target person is speaking at the current moment.

[0118] Step S260, determining that the target person is not speaking at the current moment.

[0119] Therefore, the present application can basically eliminate the negative impact of the disturbance of key points of facial images on the speech behavior detection operation and improve the problem of missed / false detection of speech behavior by executing the above steps S210 to S260, based on the combination of lip distance change detection and voting mechanism of continuous multi-frame facial images, thereby improving the accuracy of speech behavior detection, ensuring that the final output speech behavior detection results can meet the high precision requirements and effectively reflect the actual speaking status of the interacting personnel.

[0120] Optionally, see Figure 8 , Figure 8This is the second flow chart of the speaking behavior detection method provided in the embodiment of the present application. In the embodiment of the present application, due to the interference of the external dynamic environment, the interacting person will usually have unreasonable facial movement with excessive amplitude within a certain period of time, which will cause the facial key points detected by the facial key point detection model within the period of time to drift, and then cause the measured lip distance size at different times within the period of time to have some data that cannot represent the actual lip movement of the interacting person. Therefore, before further confirming whether the target person is actually in the speaking state at the current moment, it is necessary to perform anti-interference processing on the unreasonable movement phenomenon to avoid the false detection problem of non-user interaction caused by the external dynamic environment as much as possible. At this time, Figure 2 Compared with the speech behavior detection method shown in Figure 8 The speaking behavior detection method shown may further include step S270 between step S220 and step S230, wherein step S270 is performed when step S220 determines that the actual speaking trend state of the target person at the current moment is a suspected speaking state.

[0121] Step S270, detecting whether there is any unreasonable movement of the target person in historical facial images of a plurality of consecutive historical moments before the current moment based on the target facial image.

[0122] In this embodiment, the unreasonable movement phenomenon includes unreasonable up, down, left, and right movement phenomenon and unreasonable forward and backward movement phenomenon; when the robot control device 10 determines that the speaking trend state of the target person is a suspected speaking state at any time, the facial image of the target person at that moment can be used as a judgment basis for movement rationality detection, and the character images of the target person at the first preset number of consecutive historical moments before that moment are compared with the character image at that moment for face frame distribution, so as to determine whether the facial part of the corresponding target person has unreasonable up, down, left, and right movement phenomenon and / or unreasonable forward and backward movement phenomenon within the first time period with that moment as the upper time limit boundary.

[0123] When the robot control device 10 determines that there is unreasonable movement in the historical facial images of the target person at multiple consecutive historical moments before the current moment (i.e., the first preset number of historical moments), it indicates that the unreasonable movement caused by the external dynamic environment has seriously affected the lip distance reliability of the target person within the first time period. In order to improve the false detection problem of non-user interaction caused by the external dynamic environment, it can be determined that the target person is deemed to be not speaking at the current moment. The robot control device 10 will jump back to the step S210 to continue execution, so as to perform speaking behavior detection on the facial image of the target person at the next moment after the current moment (i.e., the new current moment).

[0124] When the robot control device 10 determines that there is no unreasonable movement phenomenon in the historical facial images of the target person at multiple consecutive historical moments before the current moment, it means that the facial movement phenomenon caused by the external dynamic environment has almost no effect on the lip distance reliability of the target person within the first time period. At this time, the robot control device 10 will continue to execute steps S230 to S260 to further confirm whether the target person is actually in a speaking state at the current moment.

[0125] Optionally, see Fig. 9 , Fig. 9 yes Figure 8 In the embodiment of the present application, the step S270 may include sub-steps S271 to S274 to effectively detect whether there is unreasonable movement in the historical face images of the target person at multiple consecutive historical moments before the current moment.

[0126] Sub-step S271, based on the face frame position information and face frame size information of the actual face frame of the target face image, construct a legal face frame moving area that covers the actual face frame and whose corresponding area size is larger than the face frame size information; and based on the actual area value of the actual face frame, construct a legal face frame area interval that includes the actual area value.

[0127] In this embodiment, the legal face frame moving area is used to detect whether the face part of the target person moves unreasonably up, down, left, and right in the first time period, and the legal face frame area interval is used to detect whether the face part of the target person moves unreasonably forward and backward in the first time period. Wherein, the legal face frame moving area can be obtained by taking the center position of the rectangular frame of the actual face frame of the target face image as the scaling center point, and enlarging the height and width of the actual face frame in proportion according to a preset ratio value, wherein the preset ratio value is greater than 1 (for example, the preset ratio value is 1.2); the interval lower limit value of the legal face frame area interval can be obtained by dividing the actual area value by a preset coefficient, and the interval upper limit value of the legal face frame area interval can be obtained by multiplying the actual area value by a preset coefficient, wherein the preset coefficient is greater than 1 (for example, the preset coefficient is 1.44).

[0128] Sub-step S272, for each historical face image among the historical face images at multiple historical moments, detect whether the historical face frame of the historical face image at least partially exceeds the legal face frame moving area, and detect whether the face frame area value of the historical face frame of the historical face image exceeds the legal face frame area range.

[0129] In this embodiment, for any historical face image, if at least one of the four vertex positions of the historical face frame of the historical face image is not within the legal face frame moving area, it can be determined that the historical face frame of the historical face image at least partially exceeds the legal face frame moving area.

[0130] Sub-step S273, when it is detected that among the historical face images of multiple historical moments, the historical face frame of any historical face image at least partially exceeds the legal face frame movement area, and / or when it is detected that among the historical face images of multiple historical moments, the face frame area value of any historical face image exceeds the legal face frame area range, it is determined that there is unreasonable movement of the target person in the historical face images of multiple historical moments.

[0131] Among them, when the historical face frame of any historical face image among the historical face images of the multiple historical moments at least partially exceeds the legal face frame movement area, it indicates that the target face has experienced unreasonable up, down, left, and right movement within the first time period corresponding to the multiple historical moments; when the face frame area value of any historical face image among the historical face images of the multiple historical moments exceeds the legal face frame area range, it indicates that the target face has experienced unreasonable forward and backward movement within the first time period corresponding to the multiple historical moments.

[0132] Sub-step S274, when it is detected that the historical face frames of the historical face images of multiple historical moments do not exceed the legal face frame movement area, and the corresponding face frame area values ​​do not exceed the legal face frame area range, it is determined that there is no unreasonable movement of the target person in the historical face images of multiple historical moments.

[0133] Therefore, the present application can effectively detect whether there is unreasonable movement of the target person in the historical facial images of multiple consecutive historical moments before the current moment by executing the above sub-steps S271 to S274.

[0134] The present application can, through the coordination between the above-mentioned step S270 and steps S230 to S260, perform anti-interference processing on unreasonable movement phenomena before confirming whether the target person is actually in a speaking state at the current moment, so as to avoid the false detection problem of non-user interaction caused by the external dynamic environment as much as possible.

[0135] Optionally, see Fig.10 , Fig.10 This is the third flow chart of the speech behavior detection method provided in the embodiment of the present application. Figure 2 or Figure 8 Compared with the speech behavior detection method shown in Fig.10 The speaking behavior detection method shown may also include steps S280 and S290 to ensure that the trained facial key point detection model can focus more on accurately locating the lip key points during the facial key point detection process, thereby facilitating the high-precision facial lip distance calculation using the trained facial key point detection model.

[0136] Step S280, obtaining a labeled training sample set for the facial key point detection model, wherein the labeled training sample set consists of a first facial sample image set labeled with all facial key points and a second facial sample image set labeled with only lip key points.

[0137] In this embodiment, the key point detection model trained by using face sample images with only lip key points annotated is prone to unstable lip key point prediction when facing face images affected by external factors, while the key point detection model trained by using face sample images with all face key points annotated mainly focuses on the global correlation of the face when facing face images affected by external factors, and cannot achieve more refined positioning of the lip key points. Therefore, in order to ensure that the trained face key point detection model can focus more on accurately locating the lip key points during the face key point detection process, a face sample image with all face key points annotated (for example, Figure 4 The first face sample image set with 98 face key points in the image and the first face sample image set with only lip key points annotated (for example, Figure 4 The labeled training sample set is constructed by mixing the first face sample image set with the facial key points 76-95 in the rectangular frame, wherein the plurality of sample face images included in the first face sample image set are all labeled with 98 facial key points, and the plurality of sample face images included in the second face sample image set are all labeled with 20 lip key points. In one implementation of this embodiment, the model architecture of the facial key point detection model can be implemented using a lightweight key point prediction network Shufflenetv2.

[0138] Step S290, alternately input the first face sample image set and the second face sample image set into the face key point detection model, and train the face key point detection model according to the target loss weight vector that matches the input face sample image set, with the purpose of minimizing the target loss function that matches the input face sample image set, to obtain a face key point detection model that meets the expected performance state.

[0139] In this embodiment, different face sample image sets each correspond to a different loss function, wherein the loss function corresponding to the first face sample image set may be, but is not limited to, a mean square error loss function, an L2 loss function, an L1 loss function, a Smooth L1 loss function, a huber loss function, a relative entropy loss function, a cross entropy loss function, a softmax loss function, a Focal loss loss function, and the loss function corresponding to the second face sample image set may be a WingLoss loss function; different face sample image sets each correspond to a different loss weight vector, and a first loss weight vector matching the first face sample image set and a second loss weight vector matching the second face sample image set maintain the same vector dimension (for example, 98), wherein each loss weight vector records the loss weight values ​​of 98 facial key points during the model training process; the loss weight value of the inner lip key point in the first loss weight vector is greater than the loss weight values ​​of other facial key points other than the inner lip key points, and the loss weight values ​​of other facial key points other than the inner lip key points in the first loss weight vector remain consistent (for example, the first loss weight vector can be expressed as [1,1,1,...1,1,5,5,5,1,5,5,1,1] 98 , where facial key points 89-91 and 93-95 are 6 inner lip key points, the loss weight value of each of the 6 inner lip key points is 5, and the loss weight value of each of the remaining 92 other facial key points is 1); the loss weight values ​​of other facial key points other than lip key points in the second loss weight vector are zero, and the loss weight values ​​of the inner lip key points in the second loss weight vector are greater than the loss weight values ​​of other lip key points other than inner lip key points (for example, the second loss weight vector can be expressed as [0,0,0,...1,1,5,5,5,1,5,5,5,0,0] 98 , where face key points 76-95 are lip key points, face key points 89-91 and 93-95 are 6 inner lip key points, face key points 76-88, 92 and 96-97 are 14 other lip key points, where the loss weight value of the 6 inner lip key points is 5, the loss weight value of the 14 other lip key points is 1, and the loss weight value of the remaining 78 other face key points is 0).

[0140] Therefore, the present application can ensure that the trained facial key point detection model can focus more on accurately locating the lip key points during the facial key point detection process by executing the above steps S280 and S290, thereby facilitating the high-precision calculation of the face-lip distance using the trained facial key point detection model.

[0141] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0143] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a readable storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device as the robot control device 10) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0144] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting speech behavior, characterized in that: The method comprises: Obtaining a target face image of a target person at a current moment, and calculating an actual lip distance of the target person in the target face image; Determining whether the actual speaking trend state of the target person at the current moment is a suspected speaking state according to the actual lip distance and the historical lip distance of the target person in historical facial images of a plurality of consecutive historical moments before the current moment; In the case where the actual speaking trend state is a suspected speaking state, status statistics are performed on the actual speaking trend state and the historical speaking trend states of the target person at a plurality of consecutive historical moments before the current moment, to obtain a real-time status distribution ratio of the suspected speaking state at the current moment; Detecting whether the real-time status distribution ratio exceeds a preset status ratio threshold; If it is detected that the real-time state distribution ratio exceeds the preset state ratio threshold, it is determined that the target person is speaking at the current moment; otherwise, it is determined that the target person is not speaking at the current moment.

2. The method according to claim 1, characterized in that The step of calculating the actual lip distance of the target person in the target face image comprises: Calling a pre-trained facial key point detection model to perform facial key point detection on the target facial image to obtain lip key point information in the target facial image; Calculate the upper and lower inner lip distances corresponding to each of the multiple groups of inner lip key points in the target face image according to the lip key point information, wherein each group of inner lip key points consists of an upper inner lip key point and a lower inner lip key point; The upper and lower inner lip distances corresponding to the multiple groups of inner lip key points are averaged to obtain the actual lip distance of the target person in the target face image.

3. The method according to claim 1, characterized in that The step of determining whether the actual speaking trend state of the target person at the current moment is a suspected speaking state based on the actual lip distance and the historical lip distance of the target person in historical face images of a plurality of consecutive historical moments before the current moment comprises: Calculate the effective lip distance change amplitude value of the target person at the current moment according to the actual lip distance and the historical lip distance of the target person in the historical face images of a plurality of consecutive historical moments before the current moment; Detecting whether the effective lip distance change amplitude value exceeds a preset change amplitude threshold; If it is detected that the effective lip distance change amplitude value exceeds the preset change amplitude threshold, it is determined that the actual speaking trend state of the target person at the current moment is a suspected speaking state; otherwise, it is determined that the actual speaking trend state of the target person at the current moment is a non-speaking state.

4. The method according to claim 3, characterized in that The step of calculating the effective lip distance variation amplitude value of the target person at the current moment according to the actual lip distance and the historical lip distance of the target person in the historical face images of a plurality of consecutive historical moments before the current moment comprises: Calculating the absolute value of the lip distance difference between the actual lip distance and the historical lip distances of each of the multiple historical moments; The maximum lip distance difference absolute value is selected from the lip distance difference absolute values ​​corresponding to the multiple historical moments, and the maximum lip distance difference absolute value is used as the effective lip distance change amplitude value of the target person at the current moment.

5. The method according to claim 3, characterized in that: The step of calculating the effective lip distance variation amplitude value of the target person at the current moment according to the actual lip distance and the historical lip distance of the target person in the historical face images of a plurality of consecutive historical moments before the current moment comprises: Calling a pre-trained facial key point detection model to determine the distribution ratio of the first image matching the side face posture of the target person in the target face image and the historical face images of the multiple historical moments, and the distribution ratio of the second image matching the lip area occlusion phenomenon; Detecting whether the first image distribution ratio exceeds a first preset image ratio threshold, and detecting whether the second image distribution ratio exceeds a second preset image ratio threshold; When it is detected that the first image distribution ratio exceeds the first preset image ratio threshold, and / or when it is detected that the second image distribution ratio exceeds the second preset image ratio threshold, directly setting the effective lip distance change amplitude value of the target person at the current moment to zero; When it is detected that the first image distribution ratio does not exceed the first preset image ratio threshold, and when it is detected that the second image distribution ratio does not exceed the second preset image ratio threshold, the absolute value of the lip distance difference between the actual lip distance and the historical lip distance at each of the multiple historical moments is calculated, and the maximum absolute value of the lip distance difference is selected as the effective lip distance change amplitude value of the target person at the current moment.

6. The method according to claim 1, characterized in that Before executing the step of collecting status statistics of the actual speaking trend status and the historical speaking trend status of the target person at a plurality of consecutive historical moments before the current moment, the method further includes: In the case where the actual speaking trend state is a suspected speaking state, detecting whether there is unreasonable movement of the target person in historical facial images of a plurality of consecutive historical moments before the current moment according to the target facial image; When it is detected that there is no unreasonable movement phenomenon in the historical facial images of the target person at multiple consecutive historical moments before the current moment, continue to execute the step of performing status statistics of the actual speaking trend state and the historical speaking trend state of the target person at multiple consecutive historical moments before the current moment.

7. The method according to claim 6, characterized in that The step of detecting whether there is unreasonable movement of the target person in historical facial images of a plurality of consecutive historical moments before the current moment based on the target facial image comprises: According to face frame position information and face frame size information of an actual face frame of the target face image, construct a legal face frame moving area that covers the actual face frame and has a corresponding area size larger than the face frame size information, and according to an actual area value of the actual face frame, construct a legal face frame area interval that includes the actual area value; For each historical face image among the multiple historical moments, detecting whether a historical face frame of the historical face image at least partially exceeds the legal face frame moving area, and detecting whether a face frame area value of the historical face frame of the historical face image exceeds the legal face frame area interval; When it is detected that a historical face frame of any one of the historical face images at the multiple historical moments at least partially exceeds the legal face frame movement area, and / or when it is detected that a face frame area value of any one of the historical face images at the multiple historical moments exceeds the legal face frame area interval, it is determined that the target person has unreasonable movement in the historical face images at the multiple historical moments; When it is detected that the historical face frames of each of the historical face images at the multiple historical moments do not exceed the legal face frame movement area, and the corresponding face frame area values ​​do not exceed the legal face frame area range, it is determined that there is no unreasonable movement of the target person in the historical face images at the multiple historical moments.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtain a labeled training sample set for a facial key point detection model, wherein the labeled training sample set consists of a first facial sample image set labeled with all facial key points and a second facial sample image set labeled with only lip key points; The first face sample image set and the second face sample image set are alternately input into the face key point detection model, and the face key point detection model is trained according to the target loss weight vector matched with the input face sample image set, with the purpose of minimizing the target loss function matched with the input face sample image set, so as to obtain the face key point detection model that meets the expected performance state; wherein, the first loss weight vector matched with the first face sample image set and the second loss weight vector matched with the second face sample image set maintain the same vector dimension, the loss weight value of the inner lip key point in the first loss weight vector is greater than the loss weight values ​​of other face key points other than the inner lip key points, the loss weight values ​​of other face key points other than the lip key points in the second loss weight vector are zero, and the loss weight value of the inner lip key point in the second loss weight vector is greater than the loss weight values ​​of other lip key points other than the inner lip key points.

9. A robot control device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the speaking behavior detection method according to any one of claims 1 to 8.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer device, the speaking behavior detection method according to any one of claims 1 to 8 is implemented.

Images