Robot dialogue method and device based on adaptive vision technology

Abstract

The invention discloses a robot dialogue method and device based on a self-adaptive vision technology. The robot dialogue method comprises the following steps: continuously sending an image acquisition instruction to a scene in a shooting range according to a preset period; acquiring a time node triggered by the image acquisition instruction, and adaptively matching corresponding shooting parameters; shooting a scene in response to the image acquisition instruction based on the shooting parameters to obtain a scene image; carrying out image quality evaluation on the scene image, and if the evaluation requirement is not met, polling a new shooting parameter to carry out quality evaluation again; if the evaluation requirement is met, face recognition is carried out on the scene image, if a face area is included, a voice recognition instruction is triggered and started, and after the voice recognition instruction is started, voice input by a user is obtained in real time; and converting user input voice into a text, calling the large language model to perform semantic understanding, and obtaining feedback output of the large language model. According to the method, the problem of too fast power consumption caused by simultaneous activity of the two processes is avoided, and the effectiveness of visual detection is ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a robot dialogue method and apparatus based on adaptive vision technology. Background Technology

[0002] While artificial intelligence technology has achieved a "brain leap" like ChatGPT, enabling machines to truly perceive and act like humans still requires overcoming the final hurdle of "infusing the soul into the body." Embossed intelligent robots are driving the transition from "digital intelligence" to "physical intelligence" through the deep coupling of AI algorithms and mechanical bodies. Currently, large models endow robots with the ability to understand instructions, and multimodal perception systems enable environmental interaction. In the future, with decreasing hardware costs and algorithmic evolution, general-purpose robots will move from laboratories to factories and ultimately to homes. The widespread adoption of embodied intelligent robots will also change traditional human-computer interaction models, bringing entirely new user experiences. Interacting with and controlling robots through natural language dialogue has long been a human dream. With rapid technological advancements, conversations with various devices and robots via natural language have become widespread and applied. From a human-computer interaction perspective, further improving the user experience of natural language human-computer interaction is a direction that both academia and engineering are striving towards.

[0003] Existing human-computer interaction methods mainly include the following two approaches:

[0004] 1. The wake word method: This is the most common method. A process on the device continuously listens to the user's voice input. When the user enters a specific word, this process starts continuously listening for the user's voice input. However, the wake word method is actually a rather unnatural interaction method. When we are looking at the robot, we still need to use a wake word to wake it up and interact with it. This method results in a less natural user experience and doesn't match actual human interaction.

[0005] 2. The vision + wake word approach is widely used in autonomous driving. When a person's face is continuously focused on the screen, the machine automatically detects and recognizes the face, then receives the user's voice input. Simultaneously, voice interaction can be triggered via a wake word. While this approach offers a natural and user-friendly interaction method, the vision aspect is problematic. Under different lighting conditions and with the same shooting parameters, the resulting facial images can vary significantly. Especially in strong or low light conditions, a completely white or black image might be captured, making facial recognition impossible. Furthermore, the continuous detection using both vision and voice processes is power-intensive, potentially causing the robot's battery to deplete too quickly and affecting its operating time.

[0006] Therefore, how to provide a robot dialogue method and device that avoids excessive power consumption during both visual and voice processes while effectively ensuring the effectiveness of visual detection is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a robot dialogue method and apparatus based on adaptive vision technology, which avoids the problem of excessive power consumption caused by two processes being active at the same time; at the same time, when taking pictures of faces with vision, high-quality pictures can be obtained under different lighting conditions, ensuring the effectiveness of visual detection.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] This invention first proposes a robot dialogue method based on adaptive vision technology, comprising the following steps:

[0010] S1: Continuously sends image acquisition commands to the scene within the shooting range according to the preset cycle;

[0011] S2: Obtain the time node that triggers the image acquisition command, and adaptively match the corresponding shooting parameters according to the time node;

[0012] S3: Based on the shooting parameters, respond to the image acquisition command to capture the scene and obtain a scene image;

[0013] S4: Perform image quality assessment on the scene image. If it does not meet the assessment requirements, poll for new shooting parameters and re-execute S4. If it meets the assessment requirements, perform face recognition on the scene image to determine whether the scene image contains a face region. If yes, proceed to S5. If no, return to S1.

[0014] S5: Trigger the voice recognition start command. After the voice recognition start command is activated, the user's voice input is acquired in real time.

[0015] S6: Convert the user's voice input into text, call the large language model for semantic understanding, and obtain the feedback output of the large language model.

[0016] Preferably, S1 includes: setting a timer with a preset period; when the timer is triggered, calling the shooting device interface and issuing an image acquisition command.

[0017] Preferably, in step S2, the step of adaptively matching the corresponding shooting parameters according to the time node includes:

[0018] The day is divided into different time intervals based on the intensity of light.

[0019] Set shooting parameters for different light intensity levels;

[0020] A shooting parameter of a certain light intensity level is matched for each time interval, and it is stored as the shooting device's built-in knowledge in the shooting device parameter configuration;

[0021] The system adaptively matches the corresponding shooting parameters based on the storage relationship between the time interval and the shooting parameters.

[0022] Preferably, different shooting parameters of different levels are matched for different time intervals.

[0023] Preferably, the shooting parameters include camera exposure parameters.

[0024] Preferably, in step S3, the step of shooting the scene in response to the image acquisition command based on the shooting parameters includes: automatically responding to the image acquisition command after matching the corresponding shooting parameters, and triggering shooting to obtain a scene image.

[0025] Preferably, in step S4, the scene image is evaluated for image quality. If it does not meet the evaluation requirements, the shooting parameters are automatically polled to obtain the next shooting parameters, and the scene image is evaluated for image quality based on the next shooting parameters.

[0026] Preferably, in step S4, a shooting status label is assigned to the image acquisition command; the scene image is evaluated for image quality; if it meets the evaluation requirements, the shooting status label is updated to OK; after being updated to OK, face recognition and image quality evaluation are performed simultaneously on the scene images acquired in subsequent cycles until step S4 is entered, triggering the start of the voice recognition command.

[0027] Preferably, the step of triggering the voice recognition start command in S5 includes: automatically turning on the voice pickup interface and starting the voice monitoring process to acquire the user's input voice.

[0028] The present invention further provides a robot dialogue device according to the aforementioned robot dialogue method based on adaptive vision technology, comprising:

[0029] The camera manager is configured to continuously issue image acquisition commands to the scene within the shooting range according to a preset cycle;

[0030] The adaptive parameter manager is configured to obtain the time node triggered by the image acquisition command and adaptively match the corresponding shooting parameters according to the time node.

[0031] The shooting device is configured to capture images of the scene in response to the image acquisition command based on the shooting parameters, thereby obtaining scene images;

[0032] The image quality assessment module is configured to assess the image quality of the scene image. If the image does not meet the assessment requirements, the adaptive parameter manager is invoked, and the image quality is assessed based on the new shooting parameters obtained through polling. If the image meets the assessment requirements, the face recognition module is invoked.

[0033] The face recognition module is configured to perform face recognition on the scene image, determine whether the scene image contains a face region, and if so, call the voice recognition management module; otherwise, call the image capture manager to issue an image acquisition command.

[0034] The voice recognition management module is configured to trigger a voice recognition start command, and after the voice recognition start command is activated, it acquires the user's voice input in real time.

[0035] The large language model interaction module is configured to interact with the large language model, convert the user's input speech into text, call the large language model for semantic understanding, and obtain the feedback output of the large language model.

[0036] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a robot dialogue method and device based on adaptive vision technology, which has the following beneficial effects:

[0037] This invention uses visual technology for face detection. Once a face is detected, the voice input detection process is started. This avoids the problem of excessive power consumption caused by two processes being active at the same time, thus saving robot energy consumption and extending the robot's usage time for the same battery.

[0038] When taking facial photos using vision, this invention automatically controls the camera's exposure parameters according to different lighting conditions, thereby enabling the acquisition of high-quality images under various lighting conditions. This ensures the effectiveness of visual detection, improves the quality and efficiency of human-computer interaction, and optimizes the user experience. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 The flowchart of the robot dialogue method based on adaptive vision technology provided by the present invention is shown below.

[0041] Figure 2The diagram illustrates the working principle of the robot dialogue device based on adaptive vision technology provided by this invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The first aspect of this invention discloses a robot dialogue method based on adaptive vision technology, such as... Figure 1 As shown, it includes the following steps:

[0044] S1: Continuously sends image acquisition commands to the scene within the shooting range according to the preset cycle;

[0045] S2: Obtain the time node that triggers the image acquisition command, and adaptively match the corresponding shooting parameters according to the time node;

[0046] S3: Based on the shooting parameters, respond to the image acquisition command to capture the scene and obtain the scene image;

[0047] S4: Perform image quality assessment on the scene image. If it does not meet the assessment requirements, poll for new shooting parameters and re-execute S4. If it meets the assessment requirements, perform face recognition on the scene image to determine whether the scene image contains a face region. If yes, proceed to S5. If no, return to S1.

[0048] S5: Triggers the voice recognition start command. After the voice recognition start command is activated, the user's voice input is acquired in real time.

[0049] S6: Convert user-input speech into text, call the large language model for semantic understanding, and obtain the feedback output of the large language model.

[0050] In one embodiment, S1 includes: setting a timer with a preset period; and when the timer is triggered, calling the imaging device interface to issue an image acquisition command. In this embodiment, the imaging device can be a camera configured on the robot. This is a continuously monitoring process that will continuously trigger image capture according to a fixed time period.

[0051] In practice, a timer can be started using Python or C++ code. The timer period can be set to 200ms, depending on how sensitive people are to time. When the timer is triggered, the camera interface is called to take a picture and obtain the captured photo.

[0052] In one embodiment, step S2, which involves adaptively matching the corresponding shooting parameters based on the time node, includes:

[0053] The day is divided into different time intervals based on the intensity of light.

[0054] Set shooting parameters for different light intensity levels;

[0055] A shooting parameter of a certain light intensity level is matched for each time interval, and it is stored as the shooting device's built-in knowledge in the shooting device parameter configuration;

[0056] The system adaptively matches the corresponding shooting parameters based on the storage relationship between the time interval and the shooting parameters.

[0057] In practice, the light intensity can be divided into five levels based on the image quality issues caused by different lighting conditions: high intensity light, strong light, normal light, weak light, and weak light.

[0058] Based on the five levels, a corresponding shooting parameter value is set for each robot's shooting device for each level. These values ​​will be stored as built-in knowledge in the shooting device parameter configuration.

[0059] After triggering the photo capture, the system obtains the current time and first looks up the corresponding light level, such as high intensity light or moderate intensity light. By default, it selects a shooting parameter from the shooting device's shooting parameter configuration based on the current time. That is, the camera parameter corresponding to each light level is listed in the light and camera parameter lookup table. You only need to look at the light level to obtain the corresponding camera parameter, set the camera according to the corresponding camera parameter, and trigger the photo capture.

[0060] In one embodiment, different time intervals are matched with unique and different levels of shooting parameters.

[0061] In practice, the relationship between different levels of light intensity and time is as follows:

[0062] High intensity light: 12 PM - 2 PM during the day;

[0063] Stronger light: 2 PM - 4 PM during the day;

[0064] Normal light: 10:00-12:00 and 16:00-17:00 during the day;

[0065] Low light: 7 AM - 10 AM, 5 PM - 6 PM;

[0066] Dim light: Other time periods.

[0067] In one embodiment, the shooting parameters include camera exposure parameters.

[0068] In one embodiment, S3, the step of shooting the scene based on the shooting parameters in response to the image acquisition command includes: after matching the corresponding shooting parameters, automatically calling the camera adapter to respond to the image acquisition command and triggering the shooting to obtain the scene image.

[0069] In one embodiment, in S4, the scene image is evaluated for image quality. If it does not meet the evaluation requirements, the shooting parameters are automatically polled to obtain the next shooting parameters until a suitable exposure parameter is selected. Then, the scene is captured immediately based on the suitable shooting parameters in response to the image acquisition command, and the scene image is obtained.

[0070] It is understandable that appropriate shooting parameters are those that meet the requirements for image quality assessment.

[0071] In one embodiment, in S4, a shooting status label is assigned to the image acquisition command; the scene image is evaluated for image quality; if it meets the evaluation requirements, the shooting status label is updated to OK; after being updated to OK, face recognition and image quality evaluation are performed on the scene images acquired in subsequent cycles, until entering S4 and triggering the start of the voice recognition command.

[0072] In one embodiment, the face recognition step in S4 is mainly responsible for identifying whether a face exists based on a scene image. If a face exists, it means that a user is on a call. At this time, speech recognition will be automatically started to obtain the user's voice input. Otherwise, it will be ignored.

[0073] In one embodiment, the step of triggering the start speech recognition command in S5 includes: automatically turning on the speech pickup interface and starting the speech monitoring process to acquire user input speech.

[0074] In practice, upon facial recognition, the microphone is automatically activated, and a voice monitoring process is initiated to acquire user input. Through the microphone adapter, the microphone interface is called to activate the microphone, continuously acquiring input voice data, converting it into text, and then using a large language model for semantic understanding, obtaining the model's feedback output.

[0075] In one embodiment, the large language model can be an external model used to identify user intent based on text, understand user intent, and generate large language model output data.

[0076] In one embodiment, the method further includes the following step: playing the feedback output of the large language model by calling the speaker interface, thereby realizing voice interaction with the user.

[0077] The second aspect of this invention also discloses a robot dialogue device based on an adaptive vision technology-based robot dialogue method according to the first aspect of the invention, comprising:

[0078] The camera manager is configured to continuously issue image acquisition commands to the scene within the shooting range according to a preset cycle;

[0079] The adaptive parameter manager is configured to obtain the time node triggered by the image acquisition command and adaptively match the corresponding shooting parameters according to the time node.

[0080] The shooting device is configured to capture images of the scene based on shooting parameters and image acquisition commands, thereby obtaining scene images;

[0081] The image quality assessment module is configured to assess the image quality of scene images. If the image does not meet the assessment requirements, the adaptive parameter manager is invoked, and the image quality is assessed based on the new shooting parameters obtained through polling. If the image meets the assessment requirements, the face recognition module is invoked.

[0082] The face recognition module is configured to perform face recognition on scene images, determine whether the scene image contains a face region, and if so, call the voice recognition management module; otherwise, call the image capture manager to issue an image acquisition command.

[0083] The speech recognition management module is configured to trigger a speech recognition start command. After the speech recognition start command is activated, the user's voice input is acquired in real time.

[0084] The large language model interaction module is configured to interact with the large language model, convert user input speech into text, call the large language model for semantic understanding, and obtain feedback output from the large language model.

[0085] like Figure 2 As shown, the entire system is divided into three main parts: image management and adaptive parameter management, face recognition and voice recognition management, and voice-based human-computer interaction. These three parts are connected and transmit data at the hardware level through camera adapters, microphone adapters, and speaker adapters.

[0086] The photo manager initiates a timed photo-taking task. The adaptive parameter manager automatically selects appropriate camera exposure parameters based on current lighting conditions and applies these parameters to the actual shot. The image quality assessment module evaluates the quality of the captured photo to ensure image clarity meets facial recognition requirements. If the image quality assessment module confirms the photo quality is acceptable, it passes the photo to the facial recognition module. The facial recognition module analyzes the photo to determine the presence of a face and passes the result to the speech recognition management module, instructing it to start voice monitoring. The speech recognition management module controls the microphone adapter to start recording and passes the recorded voice data to the large language model interaction module. First, the data is passed to the voice-based human-computer interaction module, which uses speech recognition technology to convert the speech into text and then uses the large model service interaction module to understand the user's intent. The large model service interaction module generates a response text and passes it to the speaker adapter for playback through the speaker.

[0087] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0088] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot dialogue method based on adaptive vision technology, characterized in that, Includes the following steps: S1: Continuously sends image acquisition commands to the scene within the shooting range according to the preset cycle; S2: Obtain the time node that triggers the image acquisition command, and adaptively match the corresponding shooting parameters according to the time node; S3: Based on the shooting parameters, respond to the image acquisition command to capture the scene and obtain a scene image; S4: Perform image quality assessment on the scene image. If it does not meet the assessment requirements, poll for new shooting parameters and re-execute S4. If it meets the assessment requirements, perform face recognition on the scene image to determine whether the scene image contains a face region. If yes, proceed to S5. If no, return to S1. S5: Trigger the voice recognition start command. After the voice recognition start command is activated, the user's voice input is acquired in real time. S6: Convert the user's voice input into text, call the large language model for semantic understanding, and obtain the feedback output of the large language model.

2. The robot dialogue method based on adaptive vision technology according to claim 1, characterized in that, S1 includes: setting a timer with a preset period; when the timer is triggered, calling the shooting device interface and issuing an image acquisition command.

3. The robot dialogue method based on adaptive vision technology according to claim 1, characterized in that, In step S2, the step of adaptively matching the corresponding shooting parameters according to the time node includes: The day is divided into different time intervals based on the intensity of light. Set shooting parameters for different light intensity levels; A shooting parameter of a certain light intensity level is matched for each time interval, and it is stored as the shooting device's built-in knowledge in the shooting device parameter configuration; The system adaptively matches the corresponding shooting parameters based on the storage relationship between the time interval and the shooting parameters.

4. The robot dialogue method based on adaptive vision technology according to claim 3, characterized in that, Different shooting parameters of different levels are matched to different time intervals.

5. A robot dialogue method based on adaptive vision technology according to claim 3, characterized in that, The shooting parameters include camera exposure parameters.

6. The robot dialogue method based on adaptive vision technology according to claim 1, characterized in that, In step S3, the step of taking a picture of the scene based on the shooting parameters in response to the image acquisition command includes: after matching the corresponding shooting parameters, automatically responding to the image acquisition command and triggering the shooting to obtain a scene image.

7. A robot dialogue method based on adaptive vision technology according to claim 1, characterized in that, In step S4, the scene image is evaluated for image quality. If it does not meet the evaluation requirements, the shooting parameters are automatically polled to obtain the next shooting parameters, and the scene image is evaluated for image quality based on the next shooting parameters.

8. A robot dialogue method based on adaptive vision technology according to claim 1, characterized in that, In step S4, a shooting status label is assigned to the image acquisition command; the scene image is evaluated for image quality; if it meets the evaluation requirements, the shooting status label is updated to OK. After being updated to OK, face recognition and image quality evaluation are performed on the scene images acquired in subsequent cycles until step S4 is entered, triggering the start of the voice recognition command.

9. A robot dialogue method based on adaptive vision technology according to claim 1, characterized in that, The steps in S5 that trigger the start voice recognition command include: automatically turning on the voice pickup interface and starting the voice monitoring process to acquire user input voice.

10. A robot dialogue device based on an adaptive vision technology according to any one of claims 1-9, characterized in that, include: The camera manager is configured to continuously issue image acquisition commands to the scene within the shooting range at preset intervals; The adaptive parameter manager is configured to obtain the time node that triggers the image acquisition command and adaptively match the corresponding shooting parameters according to the time node. The shooting device is configured to capture images of the scene in response to the image acquisition command based on the shooting parameters, thereby obtaining scene images; The image quality assessment module is configured to assess the image quality of the scene image. If the image does not meet the assessment requirements, the adaptive parameter manager is invoked, and the image quality is assessed based on the new shooting parameters obtained through polling. If the image meets the assessment requirements, the face recognition module is invoked. The face recognition module is configured to perform face recognition on the scene image, determine whether the scene image contains a face region, and if so, call the voice recognition management module; otherwise, call the image capture manager to issue an image acquisition command. The voice recognition management module is configured to trigger a voice recognition start command, and after the voice recognition start command is activated, it acquires the user's voice input in real time. The large language model interaction module is configured to interact with the large language model, convert the user's input speech into text, call the large language model for semantic understanding, and obtain the feedback output of the large language model.

Images