Humanoid robot head orientation method and system based on audio-visual collaboration

Abstract

The invention discloses a humanoid robot head orientation method and system based on audio-visual collaboration, relates to the technical field of artificial intelligence, and solves the problems that in the prior art, shielding of a human face or poor environmental conditions are not considered, the number of available effective features of detection images and voice data is reduced, and the detection efficiency is improved. And the head orientation accuracy and efficiency are low. Time synchronization data is generated according to a detection image and a detection sound source; generating directional data after generating a detection influence coefficient according to the illumination intensity and the detection data; generating a horizontal orientation adjustment error according to the adjusted detection image, then generating an alarm signal, carrying out orientation by adopting the multi-source data at the same time, monitoring the inhibition degree of the environment on the multi-source data in real time, dynamically adjusting the combined weight among the multi-source data according to the correlation among the inhibition degrees, and orienting the horizontal orientation according to the combined weight. And then the orientation vertical direction is adjusted, and implementation is carried out step by step, so that the accuracy and efficiency of final head orientation are improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a method and system for head orientation of a humanoid robot based on audio-visual collaboration. Background Art

[0002] Robot head orientation is a critical component of robotics. Relying on advanced sensor systems, such as high-precision cameras and inertial measurement units, combined with complex algorithms and control systems, it enables robots to perceive and adjust their head posture and orientation in real time. This technology not only enhances a robot's environmental perception but also greatly improves its flexibility and accuracy in complex tasks. Whether it's target tracking, human-machine interaction, or autonomous navigation, accurate and efficient head orientation technology is essential.

[0003] The prior art (invention patent with publication number CN113910217B) discloses a method for head orientation of a humanoid robot with auditory and visual collaboration, which is characterized in that it mainly includes the following steps: first, voice signals are collected through a microphone array and the time is recorded; then, a voice recognition method is used to identify the voice and the number of microphones that collect the voice is counted; then, when the number of voice microphones is greater than 1, a candidate target position is set according to the time; then, the camera is rotated and video is collected; then, a face image frame is obtained from the video, and a face recognition method is used to obtain the face recognition result, and the number of faces is counted; then, when the number of faces is greater than 1, the face area is calculated and sorted; next, candidate target faces are set according to the ranking; then, a lip movement detection algorithm is used to realize lip movement recognition; then, when the number of faces with lip movement is greater than 1, a target position is set according to the area; finally, the head of the humanoid robot is rotated to realize orientation.

[0004] The above patent determines the detected person through voice and realizes the head orientation of the humanoid robot based on the facial data of the detected person; however, it fails to take into account the obscuration of the face or poor environmental conditions, which reduces the number of effective features available for the detection image and voice data, resulting in low accuracy and efficiency of head orientation; therefore, the robot head orientation system still needs further improvement. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art; to this end, the present application proposes a humanoid robot head orientation method and system based on audio-visual collaboration, which is used to solve the technical problem that the prior art lacks consideration of the face being occluded or the environmental conditions being poor, resulting in a reduction in the number of effective features available for detecting image and voice data, resulting in low accuracy and efficiency of head orientation.

[0006] To achieve the above objectives, the first aspect of the present application provides a humanoid robot head orientation system based on audio-visual collaboration, comprising: a data acquisition module, a data analysis module, an early warning module, and a database; the data acquisition module is electrically and / or communicatively connected to the data analysis module; the data analysis module is electrically and / or communicatively connected to the early warning module; the database is electrically and / or communicatively connected to the data acquisition module, the data analysis module, and the early warning module, respectively;

[0007] The data acquisition module acquires robot data, light intensity and detection data through a data acquisition device; the detection data includes detection images and detection sound sources;

[0008] The data analysis module generates time synchronization data based on the detection image and the detection sound source; generates a detection influence coefficient based on the light intensity and the detection data; generates orientation data based on the detection influence coefficient and the time synchronization data; adjusts the robot head parameters based on the orientation data; generates an alarm signal after generating a horizontal orientation adjustment error based on the adjusted detection image;

[0009] The early warning module: makes prompts according to the alarm signal and contacts the management personnel;

[0010] The database is used to store data of each module and store historical data required for training the model.

[0011] Through the above steps, this application realizes the time synchronization of images and sound sources, and can use multi-source data at the same time to carry out orientation operations during the orientation process; at the same time, the system will monitor the environment's suppression of various types of multi-source data in real time, and deeply analyze the correlation between different degrees of suppression, and then dynamically adjust the combination weights between multi-source data; in this way, the horizontal direction is first determined, and then the vertical direction is adjusted, adopting a step-by-step implementation strategy, which ultimately effectively improves the accuracy and efficiency of head orientation.

[0012] Furthermore, generating time synchronization data according to the detected image and the detected sound source includes:

[0013] Acquire detection images and sound sources in real time;

[0014] Extract the fundamental frequency, number of harmonics, energy change rate, rise time and low-frequency energy ratio of the detected sound source

[0015] When the fundamental frequency, the number of harmonics and the energy change rate are all within the corresponding speech parameter range, the speech sound source in the detection sound source is extracted as the influencing sound source;

[0016] Otherwise, when the rise time and the low-frequency energy ratio are both within the corresponding speech parameter range, the footstep sound source in the detection sound source is extracted as the influencing sound source;

[0017] Otherwise, the influencing sound source does not exist;

[0018] The detection image and the influencing sound source are time-synchronized and integrated into time-synchronized data; the time-synchronized data refers to the detection image and the influencing sound source at the same detection time.

[0019] This embodiment takes into account that the voice data of the detected person does not always exist. Once the voice data does not exist, other valid sound source data of the detected person can be used as one of the decision data, and the detected sound source and the detected image can be unified in time so that the decision data can be more accurate and the accuracy and efficiency of head orientation can be improved.

[0020] Furthermore, generating a detection influence coefficient according to the light intensity and the detection data includes:

[0021] Acquiring light intensity and detection data; the detection data includes detection images and detection sound sources;

[0022] Extract the number of valid pixels YXS and target pixels MXS as well as the actual distance SJ in the detection image, and obtain the height RG of the detection person in real time;

[0023] The occlusion rate function ZDF(YXS, MXS, SJ, RG) is constructed through the nonlinear relationship between the effective pixel number YXS, the target pixel number MXS, the actual distance SJ, the height of the detected person RG and the final occlusion rate;

[0024] Substitute the number of effective pixels YXS and target pixels MXS in the detection image, as well as the actual distance SJ and the height of the detection person RG into the occlusion rate function to calculate the final occlusion rate ZDL;

[0025] Extract the speech energy YN, footstep energy JN and other energy QN corresponding to the speech sound source, footstep sound source and other sound sources in the detection sound source;

[0026] The image detection influence function TJF(ZDL, GQ) is constructed through the nonlinear relationship between the final occlusion rate ZDL, the light intensity GQ and the image detection influence coefficient;

[0027] Substitute the final occlusion rate ZDL and light intensity GQ into the image detection influence function TJF(ZDL, GQ) to calculate the image detection influence coefficient;

[0028] The sound detection influence function SJF(YN, JN, QN) is constructed through the nonlinear relationship between the voice energy YN, the footstep energy JN and other energies QN and the sound detection abnormality coefficient;

[0029] Substitute the speech energy YN, footstep energy JN and other energy QN into the sound detection influence function SJF (YN, JN, QN) to calculate the sound detection influence coefficient.

[0030] This application considers multiple parameters to measure the degree of inhibition of the environment on decision data, and quantifies the degree of inhibition of the environment on several decision data based on the relationship between multiple parameters, providing accurate data support for the weight distribution between subsequent decision data, and improving the accuracy of subsequent orientation.

[0031] Furthermore, generating directional data according to the detection influence coefficient and the time synchronization data includes:

[0032] Acquire detection influence coefficients and time synchronization data; the detection influence coefficients include image detection influence coefficients and sound detection influence coefficients; the time synchronization data include detection images and influencing sound sources;

[0033] Generate a decision weight based on the detection influence coefficient; the decision weight includes an image weight and a sound source weight;

[0034] The decision weight and the time synchronization data are combined accordingly to obtain the decision data;

[0035] Inputting the decision data into a horizontal orientation decision model to obtain a horizontal orientation position; the horizontal orientation decision model is constructed by an artificial intelligence model;

[0036] Obtain the height and actual distance of the detected person in real time;

[0037] Generates vertical orientation angle based on the height of the inspector, actual distance and robot parameters.

[0038] Furthermore, generating a decision weight according to the detection influence coefficient includes:

[0039] Obtaining a detection influence coefficient; the detection influence coefficient includes an image detection influence coefficient and a sound detection influence coefficient;

[0040] The image influence ratio and the sound influence ratio are obtained by performing a difference operation between the detected influence coefficient and the corresponding influence threshold, and then performing a ratio operation with the influence threshold;

[0041] The decision weight is generated by the inverse proportional relationship between the size relationship between the image influence ratio and the sound influence ratio and their corresponding decision weights.

[0042] Furthermore, the horizontal directional decision model is constructed through an artificial intelligence model, including:

[0043] Obtaining a number of historical decision data and their corresponding historical horizontal directional positions;

[0044] Dividing a number of historical decision data and their corresponding historical horizontal directional positions into training data, verification data, and test data;

[0045] Perform data preprocessing on the training data, verification data and test data to obtain the training set, verification set and test set;

[0046] Select an artificial intelligence model as the base model;

[0047] Train the basic model using the training set, and adjust the learning rate and hyperparameters on the validation set to obtain the pre-trained model;

[0048] By verifying the pre-trained model on the test set, we finally obtain a horizontal orientation decision model whose input is decision data and output is horizontal orientation position.

[0049] Furthermore, generating a vertical orientation angle according to the height of the detection person, the actual distance and the robot parameters includes:

[0050] Real-time acquisition of the height of the inspector, the actual distance, and the robot parameters; the robot parameters include the robot height;

[0051] The vertical angle function CJF(RG, SJ, JG) is constructed by detecting the nonlinear relationship between the personnel height RG, the actual distance SJ, the robot height JG and the vertical orientation angle; wherein the vertical angle function is a trigonometric function;

[0052] Substitute the detection personnel height RG, actual distance SJ and robot height JG into the vertical angle function to calculate the vertical orientation angle.

[0053] Furthermore, generating an alarm signal after generating a horizontal orientation adjustment error according to the adjusted detection image includes:

[0054] Obtain the adjusted detection image;

[0055] Extract the head coordinates TZ (x, y) of the person in the detection image using the YOLO algorithm; extract the image size TC of the detection image; the image size includes the length corresponding to the x-axis and the height corresponding to the y-axis;

[0056] Quantify the horizontal orientation adjustment error by calculating the head coordinates and detecting the angle between the middle point of the image and the y-axis;

[0057] An alarm signal is generated according to the horizontal orientation adjustment error, the image impact ratio, and the sound impact ratio.

[0058] Furthermore, generating an alarm signal according to the horizontal orientation adjustment error, the image impact ratio, and the sound impact ratio includes:

[0059] Obtaining horizontal orientation adjustment error, image impact ratio, and sound impact ratio;

[0060] Determine whether the horizontal orientation adjustment error is within the normal error range; if yes, do nothing; if no, generate a horizontal orientation deviation warning signal;

[0061] Determine whether the image impact ratio is greater than a ratio threshold; if yes, generate an image decision environment abnormality alarm signal; if no, do nothing;

[0062] Determine whether the sound impact ratio is greater than the ratio threshold; if so, generate a sound decision noise abnormality alarm signal; if not, do nothing.

[0063] This application performs adjustment verification after orientation adjustment to determine whether there is a large deviation in the horizontal orientation position, and monitors the detection influence coefficient in real time. It can issue an alarm signal in time when the environmental conditions are harsh and the orientation has a large deviation, thereby improving the comprehensiveness of the robot head orientation system.

[0064] Another aspect of the present invention provides a humanoid robot head orientation method based on audio-visual collaboration, comprising:

[0065] S0: Acquire robot data, light intensity, and detection data; the detection data includes detection images and detection sound sources;

[0066] S1: Generate time synchronization data based on the detection image and detection sound source; generate detection influence coefficient based on light intensity and detection data;

[0067] S2: Generate orientation data based on the detection influence coefficient and time synchronization data; adjust the robot head parameters based on the orientation data;

[0068] S3: generating an adjustment error according to the adjusted detection image and then generating an alarm signal;

[0069] S4: Prompt according to the alarm signal and contact the management.

[0070] Compared with the prior art, the present invention has the following advantages:

[0071] 1. The present application generates time-synchronized data based on a detection image and a detection sound source; generates a detection influence coefficient based on light intensity and detection data; generates orientation data based on the detection influence coefficient and time-synchronized data; adjusts the robot head parameters based on the orientation data; generates an alarm signal after a horizontal orientation adjustment error is generated based on the adjusted detection image. By time-synchronizing the image and the sound source, the application can use multi-source data at the same time for orientation, and monitor the degree of suppression of the multi-source data by the environment in real time. The application can dynamically adjust the combination weights between the multi-source data based on the correlation between the suppression degrees, and use this to determine the horizontal orientation, and then adjust the vertical orientation. This is implemented step by step, so that the accuracy and efficiency of the final head orientation are improved.

[0072] 2. This application takes into account that the voice data of the detected person does not always exist. Once the voice data does not exist, other valid sound source data of the detected person can be used as one of the decision data, and the detected sound source and the detected image can be unified in time so that the decision data can be more accurate and the accuracy and efficiency of head orientation can be improved.

[0073] 3. This application divides and conquers the process of robot head orientation. First, the horizontal orientation position is obtained through a pre-trained horizontal orientation decision model based on the decision data after weight distribution. Based on the horizontal orientation position, the vertical orientation angle is generated according to the height of the detected person, the actual distance and the robot parameters, so that the final head orientation result is more accurate.

[0074] 4. This application dynamically adjusts the trends and values of several decision weights corresponding to the decision data by detecting the influence states of several influence coefficients under their corresponding influence thresholds, so that the decision data can be relatively adjusted to different degrees of weight under the detection influence coefficient to adapt to the impact of the environment on the decision data, thereby increasing the decision weights of favorable decision data in the decision data and improving the accuracy of directional calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0076] Figure 1 This is a schematic diagram of the principle of the humanoid robot head orientation system based on audio-visual collaboration of this application;

[0077] Figure 2 Generate a flow chart for the targeted data of this application;

[0078] Figure 3 This is a flow chart of the humanoid robot head orientation method based on audio-visual collaboration in this application. DETAILED DESCRIPTION

[0079] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0080] See also Figure 1 In a first aspect, an embodiment of the present application provides a humanoid robot head orientation system based on audio-visual collaboration, comprising: a data acquisition module, a data analysis module, an early warning module, and a database; the data acquisition module is electrically and / or communicatively connected to the data analysis module; the data analysis module is electrically and / or communicatively connected to the early warning module; the database is electrically and / or communicatively connected to the data acquisition module, the data analysis module, and the early warning module, respectively;

[0081] Data acquisition module: This module acquires robot data, light intensity, and detection data through data acquisition equipment. Detection data includes detection images and detection sound sources. Data acquisition equipment includes several sensors. Detection images refer to images captured by cameras and used for head orientation. Detection sound sources refer to all sound sources collected in the environment, including voice, footsteps, and other sound sources. Robot data includes robot head parameters, etc.

[0082] Data analysis module: Generates time-synchronized data based on the detection image and the detection sound source; time-synchronized data refers to the decision data required for the robot head orientation at the same time; generates a detection influence coefficient based on the light intensity and detection data, and the detection influence coefficient refers to the degree of inhibition of different decision data by the environment; generates orientation data based on the detection influence coefficient and time-synchronized data, and the orientation data includes horizontal positioning position and vertical orientation angle; adjusts the robot head parameters based on the orientation data; generates an alarm signal after generating a horizontal orientation adjustment error based on the adjusted detection image; the horizontal orientation adjustment error refers to the error angle after the horizontal orientation position adjustment;

[0083] Early warning module: issues prompts and contacts management personnel based on alarm signals; alarm signals include horizontal orientation deviation warning signals and image decision environment abnormality warning signals;

[0084] The database is used to store data for each module and store historical data required for training the model.

[0085] In this embodiment, the time synchronization data is generated according to the detected image and the detected sound source, including:

[0086] Acquire detection images and sound sources in real time;

[0087] Extract the fundamental frequency, number of harmonics, energy change rate, rise time and low-frequency energy ratio of the detected sound source;

[0088] When the fundamental frequency, number of harmonics, and energy change rate are all within their corresponding speech parameter ranges, the speech source in the detection sound source is extracted as the influencing sound source. In this embodiment, the speech source is extracted using the DSB method. The speech parameter range is set based on experience. In this embodiment, the speech parameter range corresponding to the fundamental frequency is set to 80Hz to 260Hz; the speech parameter range corresponding to the number of harmonics is set to be greater than or equal to 5; and the speech parameter range corresponding to the energy change rate is set to be greater than>30%.

[0089] Otherwise, when both the rise time and the low-frequency energy ratio are within their corresponding speech parameter ranges, the footstep sound source in the detection sound source is extracted as the influencing sound source. In this embodiment, the footstep sound source is extracted by the transient impact detection method. In this embodiment, the speech parameter range corresponding to the rise time is set to less than <10ms; and the speech parameter range corresponding to the low-frequency energy ratio is set to greater than 55%.

[0090] Otherwise, the influencing sound source does not exist; the influencing sound source refers to the sound source that can play a decisive role in head orientation. Other sound sources, such as noise, have an inhibitory effect on head orientation, not a decisive role.

[0091] The detection image and the influencing sound source are time-synchronized and integrated into time-synchronized data; the time-synchronized data refers to the detection image and the influencing sound source at the same detection time.

[0092] This embodiment fully takes into account the actual situation that the voice data of the detected person is not continuously available. When the voice data is missing, other valid sound source data of the detected person can be included in the decision data scope; at the same time, the detected sound source and the detected image are accurately aligned at the time level to ensure that the decision data is more accurate and reliable, thereby improving the accuracy and efficiency of head orientation.

[0093] In this embodiment, the detection influence coefficient is generated according to the light intensity and the detection data, including:

[0094] Obtain light intensity and detection data; the detection data includes detection images and detection sound sources;

[0095] Extract the number of valid pixels YXS and target pixels MXS as well as the actual distance SJ in the detection image, and obtain the height RG of the detection person in real time;

[0096] The occlusion rate function ZDF(YXS, MXS, SJ, RG) is constructed based on the nonlinear relationship between the effective pixel number YXS, the target pixel number MXS, the actual distance SJ, the height of the detected person RG and the final occlusion rate. The formula of the occlusion rate function is expressed as follows:

[0097] Among them, JY represents the distance threshold, which is set according to experience. When the actual distance exceeds the distance threshold, it is considered that the distance between the detection person and the robot is far away. Setting the distance threshold to correct the occlusion rate at long distances can prevent misjudgment of small targets; SJ max It is expressed as the maximum distance and is set based on experience; k is expressed as the ratio between the focal length and the pixel density calibration constant inside the camera; the specific value is set according to the design of the camera;

[0098] Substitute the number of effective pixels YXS and target pixels MXS in the detection image, as well as the actual distance SJ and the height of the detection person RG into the occlusion rate function to calculate the final occlusion rate ZDL;

[0099] Extract the speech energy YN, footstep energy JN and other energy QN corresponding to the speech sound source, footstep sound source and other sound sources in the detection sound source;

[0100] The image detection influence function TJF(ZDL, GQ) is constructed through the nonlinear relationship between the final occlusion rate ZDL, the light intensity GQ and the image detection influence coefficient;

[0101] The specific formula of the image detection influence function is as follows:

[0102] Among them, GQ optimal It is expressed as the optimum value of light intensity. The specific value is set according to experience. In this embodiment, GQ optimal The value is set to 500 lux. Light intensity deviation is quantified using the squared relative error term, which reflects the symmetrical impact of excessively bright and dark lighting on image quality in real scenes. The image detection impact coefficient increases as the final occlusion rate increases and the light intensity deviates more significantly from the optimal light intensity value.

[0103] Substitute the final occlusion rate ZDL and light intensity GQ into the image detection influence function TJF(ZDL, GQ) to calculate the image detection influence coefficient;

[0104] The sound detection influence function SJF(YN, JN, QN) is constructed through the nonlinear relationship between the voice energy YN, the footstep energy JN and other energies QN and the sound detection abnormality coefficient;

[0105] The specific formula of the sound detection influence function is as follows:

[0106] When the influencing sound source is a speech sound source, the specific formula of the sound detection influence function is expressed as follows:

[0107] When the influencing sound source is footsteps, the speech energy YN is 0; the specific formula of the sound detection influence function is expressed as follows:

[0108] Substitute the speech energy YN, footstep energy JN and other energy QN into the sound detection influence function SJF (YN, JN, QN) to calculate the sound detection influence coefficient.

[0109] See also Figure 2 In this embodiment, generating directional data based on the detection influence coefficient and time synchronization data includes:

[0110] Acquire detection influence coefficients and time synchronization data; the detection influence coefficients include image detection influence coefficients and sound detection influence coefficients; the time synchronization data includes detection images and influencing sound sources;

[0111] Generate decision weights based on the detection influence coefficient; the decision weights include image weights and sound source weights;

[0112] The decision weight and the time synchronization data are combined accordingly to obtain the decision data;

[0113] Inputting the decision data into the horizontal orientation decision model to obtain the horizontal orientation position; the horizontal orientation decision model is constructed through an artificial intelligence model;

[0114] Obtain the height and actual distance of the detected person in real time;

[0115] Generate vertical orientation angle according to the height of the detection personnel, actual distance and robot parameters; orientation data includes horizontal orientation position and vertical orientation angle.

[0116] This embodiment adopts a divide-and-conquer strategy to optimize the robot head orientation process; specifically, the horizontal orientation position is first accurately calculated based on the decision data processed by weight distribution with the help of a pre-trained horizontal orientation decision model; after obtaining the horizontal orientation position, the vertical orientation angle is further generated by combining the height of the detection person, the actual distance and the robot's own parameters; in this way, the final orientation result of the robot head is made more accurate.

[0117] In this embodiment, the decision weight is generated according to the detection influence coefficient, including:

[0118] Obtaining a detection influence coefficient; the detection influence coefficient includes an image detection influence coefficient and a sound detection influence coefficient;

[0119] By performing a difference operation between the detected influence coefficient and its corresponding influence threshold, and then performing a ratio operation with the corresponding influence threshold, a number of influence ratios YB are obtained; the influence threshold is set based on experience;

[0120] The calculation formula of the impact ratio is expressed as: Among them, j represents the number of the detection influence coefficient, JYX j Expressed as the detection influence coefficient corresponding to the jth number, YY j It is represented by the impact threshold corresponding to the jth number; the impact ratio includes the image impact ratio and the sound impact ratio; the impact ratio can be positive or negative. When the impact ratio is positive, it means that the negative impact of the environment on it is increasing;

[0121] By the formula BC=max(YB j )-min(YB j ) Calculate the proportional difference BC;

[0122] The decision weight corresponding to the maximum value in the influence ratio is:

[0123] The decision weight corresponding to the minimum value in the influence ratio is: In this embodiment, the initial decision weights are all set to 0.5, that is, it is assumed that the initial impact effects of the detection image and the influencing sound source in orientation are the same; as the impact ratio corresponding to the detection influence coefficient increases, the degree of suppression of the increase in the detection influence coefficient becomes greater, and therefore the weight coefficient corresponding to the detection data of the corresponding degree is reduced to suppress the negative impact brought by the detection data corresponding to the high detection influence coefficient.

[0124] In another embodiment, the decision weights are generated by an artificial intelligence model, including:

[0125] Obtain the detection influence coefficient and its corresponding initial weight;

[0126] Input the detection influence coefficient and its corresponding initial weight into the weight generation model to obtain a number of decision weights;

[0127] The weight generation model is constructed through a machine learning model, including:

[0128] Obtain several historical detection influence coefficients and historical weight coefficients and their corresponding historical decision weights;

[0129] Divide several historical detection influence coefficients and historical weight coefficients and their corresponding historical decision weights into training data, verification data, and test data;

[0130] Data preprocessing is performed on the training data, validation data, and test data to obtain the training set, validation set, and test set; the ratio between the training set, test set, and validation set is 7:2:1;

[0131] Select a machine learning model as the basic model; machine learning models include convolutional neural network models, etc.

[0132] Train the basic model using the training set, and adjust the learning rate and hyperparameters on the validation set to obtain the pre-trained model;

[0133] By verifying the pre-trained model on the test set, we finally obtain a weight generation model whose input is the detection influence coefficient and weight coefficient and whose output is the decision weight.

[0134] This embodiment dynamically optimizes the adjustment trends and specific values of several decision weights corresponding to the decision data based on the influence states of several detection influence coefficients within their corresponding influence threshold ranges; in this way, the decision data can achieve different degrees of weight adjustment based on the detection influence coefficient, so as to better adapt to the impact of the environment on the decision data, effectively improve the decision weight of the favorable part of the decision data, and thus improve the accuracy of the directional calculation.

[0135] The horizontal directional decision model in this embodiment is constructed using an artificial intelligence model, including:

[0136] Obtaining a number of historical decision data and their corresponding historical horizontal directional positions;

[0137] Dividing a number of historical decision data and their corresponding historical horizontal directional positions into training data, verification data, and test data;

[0138] Data preprocessing is performed on the training data, validation data, and test data to obtain the training set, validation set, and test set; the ratio between the training set, test set, and validation set is 7:2:1;

[0139] Select an artificial intelligence model as the basic model; artificial intelligence models include BP models, etc.

[0140] Train the basic model using the training set, and adjust the learning rate and hyperparameters on the validation set to obtain the pre-trained model;

[0141] By verifying the pre-trained model on the test set, we finally obtain a horizontal orientation decision model whose input is decision data and output is horizontal orientation position.

[0142] In this embodiment, the vertical orientation angle is generated according to the height of the detected person, the actual distance, and the robot parameters, including:

[0143] Real-time acquisition of the detection personnel's height, actual distance and robot parameters; robot parameters include robot height;

[0144] The vertical angle function CJF(RG, SJ, JG) is constructed by detecting the nonlinear relationship between the personnel height RG, the actual distance SJ, the robot height JG and the vertical orientation angle; wherein the vertical angle function is a trigonometric function;

[0145] The formula of the vertical angle function is:

[0146] Substitute the detection personnel height RG, actual distance SJ and robot height JG into the vertical angle function to calculate the vertical orientation angle; the vertical angle includes positive angles and negative angles. The positive angle refers to the angle at which the robot's head and eyes are adjusted downward when looking straight ahead, and the negative angle refers to the angle at which the robot's head and eyes are adjusted upward when looking straight ahead.

[0147] In this embodiment, generating an alarm signal after generating a horizontal orientation adjustment error according to the adjusted detection image includes:

[0148] Obtain the adjusted detection image;

[0149] The YOLO algorithm is used to extract the head coordinates TZ (x, y) of the person in the detection image; the image size TC of the detection image is extracted; the image size includes the length CD corresponding to the x-axis and the height GD corresponding to the y-axis;

[0150] The horizontal orientation adjustment error is quantified by calculating the angle between the head coordinates and the midpoint of the detection image and the y-axis. The horizontal orientation adjustment error represents the deviation angle of the detection person in the horizontal direction after orientation.

[0151] The calculation formula of horizontal orientation adjustment error is expressed as: The coordinates (CD / 2, GD / 2) represent the center point of the image captured by the robot head after the horizontal orientation position is adjusted. If the horizontal positioning is not accurate enough, the x-value of the coordinates of the inspector's head in the inspection image will not appear on the y-axis corresponding to the center point of the inspection image, which can be used to determine the accuracy of the horizontal orientation position.

[0152] An alarm signal is generated according to the horizontal orientation adjustment error, the image impact ratio, and the sound impact ratio.

[0153] In this embodiment, generating an alarm signal according to the horizontal orientation adjustment error, the image impact ratio, and the sound impact ratio includes:

[0154] Obtaining horizontal orientation adjustment error, image impact ratio, and sound impact ratio;

[0155] Determine whether the horizontal orientation adjustment error is within the normal error range. The normal error is set based on experience. In this embodiment, the normal error is set to 0°-5°. If yes, do nothing. If no, generate a horizontal orientation deviation warning signal.

[0156] Determine whether the image impact ratio is greater than a ratio threshold; if yes, generate an image decision environment abnormality alarm signal; if no, do nothing;

[0157] Determine whether the sound impact ratio is greater than the ratio threshold; if so, generate a sound decision noise anomaly alarm signal; if not, do nothing; the ratio threshold is set based on experience, and in this embodiment, the ratio threshold is set to 0.3; if the impact ratio is greater than the ratio threshold, the corresponding environment will have a greater inhibitory effect on the detection data, indicating that the environmental state is not conducive to head orientation.

[0158] See also Figure 3 Another aspect of the present application provides a method for head orientation of a humanoid robot based on audio-visual collaboration, comprising:

[0159] S0: Acquire robot data, light intensity, and detection data; detection data includes detection images and detection sound sources;

[0160] S1: Generate time synchronization data based on the detection image and detection sound source; generate detection influence coefficient based on light intensity and detection data;

[0161] S2: Generate orientation data based on the detection influence coefficient and time synchronization data; adjust the robot head parameters based on the orientation data;

[0162] S3: generating an adjustment error according to the adjusted detection image and then generating an alarm signal;

[0163] S4: Prompt according to the alarm signal and contact the management.

[0164] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0165] The working principle of this application is: by acquiring robot data, light intensity and detection data; generating time synchronization data according to the detection image and detection sound source; generating a detection influence coefficient according to the light intensity and detection data; generating orientation data according to the detection influence coefficient and time synchronization data; adjusting the robot head parameters according to the orientation data; generating an alarm signal after adjusting the error according to the adjusted detection image; making a prompt according to the alarm signal and contacting the management personnel. By time synchronizing the image and sound source, it is possible to use multi-source data at the same time for orientation during orientation, and at the same time monitor the degree of suppression of the environment on the multi-source data in real time, and dynamically adjust the combination weights between the multi-source data according to the correlation between the suppression degrees, and use this to determine the horizontal orientation, and then adjust the vertical direction of the orientation, and implement it step by step, so that the accuracy and efficiency of the final head orientation are improved, avoiding the problem that the existing technology lacks consideration of the face being occluded or the environmental conditions being poor, which reduces the number of effective features available for the detection image and voice data, resulting in low accuracy and efficiency of head orientation.

[0166] The above embodiments are only used to illustrate the technical method of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present application.

Claims

1. A humanoid robot head orientation system based on audio-visual collaboration, characterized by: include: Interconnected data acquisition module and data analysis module; The data acquisition module acquires robot data, light intensity and detection data through a data acquisition device; the detection data includes detection images and detection sound sources; The data analysis module generates time synchronization data based on the detection image and the detection sound source; generates a detection influence coefficient based on the light intensity and the detection data; generates orientation data based on the detection influence coefficient and the time synchronization data; adjusts the robot head parameters based on the orientation data; and generates an alarm signal after generating a horizontal orientation adjustment error based on the adjusted detection image.

2. The humanoid robot head orientation system based on audio-visual collaboration according to claim 1, characterized in that: Generating time synchronization data according to the detected image and the detected sound source includes: Acquire detection images and sound sources in real time; Extract the fundamental frequency, number of harmonics, energy change rate, rise time and low-frequency energy ratio of the detected sound source When the fundamental frequency, the number of harmonics and the energy change rate are all within the corresponding speech parameter range, the speech sound source in the detection sound source is extracted as the influencing sound source; Otherwise, when the rise time and the low-frequency energy ratio are both within the corresponding speech parameter range, the footstep sound source in the detection sound source is extracted as the influencing sound source; Otherwise, the influencing sound source does not exist; The detection image and the influencing sound source are time-synchronized and integrated into time-synchronized data; the time-synchronized data refers to the detection image and the influencing sound source at the same detection time.

3. The humanoid robot head orientation system based on audio-visual collaboration according to claim 1, characterized in that: Generating the detection influence coefficient according to the light intensity and the detection data includes: Acquiring light intensity and detection data; the detection data includes detection images and detection sound sources; Extract the number of valid pixels YXS and target pixels MXS as well as the actual distance SJ in the detection image, and obtain the height RG of the detection person in real time; The occlusion rate function ZDF(YXS, MXS, SJ, RG) is constructed through the nonlinear relationship between the effective pixel number YXS, the target pixel number MXS, the actual distance SJ, the height of the detected person RG and the final occlusion rate; Substitute the number of effective pixels YXS and target pixels MXS in the detection image, as well as the actual distance SJ and the height of the detection person RG into the occlusion rate function to calculate the final occlusion rate ZDL; Extract the speech energy YN, footstep energy JN and other energy QN corresponding to the speech sound source, footstep sound source and other sound sources in the detection sound source; The image detection influence function TJF(ZDL, GQ) is constructed through the nonlinear relationship between the final occlusion rate ZDL, the light intensity GQ and the image detection influence coefficient; Substitute the final occlusion rate ZDL and light intensity GQ into the image detection influence function TJF(ZDL, GQ) to calculate the image detection influence coefficient; The sound detection influence function SJF(YN, JN, QN) is constructed through the nonlinear relationship between the voice energy YN, the footstep energy JN and other energies QN and the sound detection abnormality coefficient; Substitute the speech energy YN, footstep energy JN and other energy QN into the sound detection influence function SJF (YN, JN, QN) to calculate the sound detection influence coefficient.

4. The humanoid robot head orientation system based on audio-visual collaboration according to claim 1, characterized in that: Generating directional data according to the detection influence coefficient and the time synchronization data includes: Acquire detection influence coefficients and time synchronization data; the detection influence coefficients include image detection influence coefficients and sound detection influence coefficients; the time synchronization data include detection images and influencing sound sources; Generate a decision weight based on the detection influence coefficient; the decision weight includes an image weight and a sound source weight; The decision weight and the time synchronization data are combined accordingly to obtain the decision data; Inputting the decision data into a horizontal orientation decision model to obtain a horizontal orientation position; the horizontal orientation decision model is constructed by an artificial intelligence model; Obtain the height and actual distance of the detected person in real time; Generates vertical orientation angle based on the height of the inspector, actual distance and robot parameters.

5. The humanoid robot head orientation system based on audio-visual collaboration according to claim 4, characterized in that: Generating a decision weight according to the detection influence coefficient includes: Obtaining a detection influence coefficient; the detection influence coefficient includes an image detection influence coefficient and a sound detection influence coefficient; The image influence ratio and the sound influence ratio are obtained by performing a difference operation between the detected influence coefficient and the corresponding influence threshold, and then performing a ratio operation with the influence threshold; The decision weight is generated by the inverse proportional relationship between the size relationship between the image influence ratio and the sound influence ratio and their corresponding decision weights.

6. The humanoid robot head orientation system based on audio-visual collaboration according to claim 4, characterized in that: The horizontal directional decision model is constructed through an artificial intelligence model, including: Obtaining a number of historical decision data and their corresponding historical horizontal directional positions; Dividing a number of historical decision data and their corresponding historical horizontal directional positions into training data, verification data, and test data; Perform data preprocessing on the training data, verification data and test data to obtain the training set, verification set and test set; Select an artificial intelligence model as the base model; Train the basic model using the training set, and adjust the learning rate and hyperparameters on the validation set to obtain the pre-trained model; By verifying the pre-trained model on the test set, we finally obtain a horizontal orientation decision model whose input is decision data and output is horizontal orientation position.

7. The humanoid robot head orientation system based on audio-visual collaboration according to claim 4, characterized in that: Generating a vertical orientation angle according to the height of the detection personnel, the actual distance, and the robot parameters includes: Real-time acquisition of the height of the inspector, the actual distance, and the robot parameters; the robot parameters include the robot height; The vertical angle function CJF(RG, SJ, JG) is constructed by detecting the nonlinear relationship between the personnel height RG, the actual distance SJ, the robot height JG and the vertical orientation angle; wherein the vertical angle function is a trigonometric function; Substitute the detection personnel height RG, actual distance SJ and robot height JG into the vertical angle function to calculate the vertical orientation angle.

8. The humanoid robot head orientation system based on audio-visual collaboration according to claim 1, characterized in that: The step of generating an alarm signal after generating a horizontal orientation adjustment error according to the adjusted detection image comprises: Obtain the adjusted detection image; Extract the head coordinates TZ (x, y) of the person in the detection image using the YOLO algorithm; extract the image size TC of the detection image; the image size includes the length corresponding to the x-axis and the height corresponding to the y-axis; Quantify the horizontal orientation adjustment error by calculating the head coordinates and detecting the angle between the middle point of the image and the y-axis; An alarm signal is generated according to the horizontal orientation adjustment error, the image impact ratio, and the sound impact ratio.

9. The humanoid robot head orientation system based on audio-visual collaboration according to claim 8, characterized in that: Generating an alarm signal according to the horizontal orientation adjustment error, the image impact ratio, and the sound impact ratio includes: Obtaining horizontal orientation adjustment error, image impact ratio, and sound impact ratio; Determine whether the horizontal orientation adjustment error is within the normal error range; if yes, do nothing; if no, generate a horizontal orientation deviation warning signal; Determine whether the image impact ratio is greater than a ratio threshold; if yes, generate an image decision environment abnormality alarm signal; if no, do nothing; Determine whether the sound impact ratio is greater than the ratio threshold; if so, generate a sound decision noise abnormality alarm signal; if not, do nothing.

10. A humanoid robot head orientation method based on audio-visual collaboration, applied to a humanoid robot head orientation system based on audio-visual collaboration according to any one of claims 1 to 9, characterized in that: include: S0: Acquire robot data, light intensity, and detection data; the detection data includes detection images and detection sound sources; S1: Generate time synchronization data based on the detection image and detection sound source; generate detection influence coefficient based on light intensity and detection data; S2: Generate orientation data based on the detection influence coefficient and time synchronization data; adjust the robot head parameters based on the orientation data; S3: generating an adjustment error according to the adjusted detection image and then generating an alarm signal.

Images