A multi-modal intelligent man-machine interaction control method and system for an unmanned sanitation vehicle

Abstract

The application discloses a kind of unmanned sanitation vehicle multi-modal intelligent man-machine interaction control method and system, it is related to unmanned sanitation vehicle and man-machine interaction technical field.The method includes: continuously obtaining the visual data, voice data, positioning data and vehicle state data of the periphery of vehicle, identifies interactive intention;When meeting the preset activation condition, control vehicle enters interactive parking state;Voice data is enhanced and identified, and interactive text is obtained;Analysis result is obtained by local sanitation knowledge base or cloud processing module, and reply content is generated and output in combination with vehicle state data;When meeting the interactive end condition, restore preset working state.Thereby forming the control closed loop of intention identification, parking linkage, collaborative response and job recovery, improve pedestrian interaction identification accuracy, voice interaction stability under complex environment and the cooperativity of man-machine interaction and vehicle safety control.

Description

Technical Field

[0001] This application relates to the field of unmanned sanitation vehicles and human-machine interaction technology, and in particular to a multimodal intelligent human-machine interaction control method and system for unmanned sanitation vehicles. Background Technology

[0002] With the development of autonomous driving technology, environmental perception technology, and vehicle drive-by-wire technology, driverless sanitation vehicles are gradually being applied to sweeping, cleaning, and garbage transfer operations in environments such as parks, communities, and municipal roads. During operation, driverless sanitation vehicles typically need to continuously travel in open or semi-open areas frequented by pedestrians, non-motorized vehicles, and other traffic participants. Therefore, in addition to completing autonomous driving and sanitation operations, they also need to provide appropriate feedback based on the behavior and expressions of people around the vehicle to provide necessary information prompts, operational consultations, and safety interactions.

[0003] Currently, information exchange between unmanned sanitation vehicles and people outside the vehicle mostly relies on one-way interaction methods such as preset voice broadcasts, displayed information, or light prompts. These methods often struggle to accurately determine whether a pedestrian has an interaction need based on facial orientation, gaze, posture, or voice, leading to delayed responses or accidental triggering. While some vehicles are equipped with voice interaction functions, they typically rely on pre-configured wake-up methods and fixed command sets. In complex environments such as outdoor noise, vehicle operation sounds, and multiple people talking, the stability of voice acquisition and semantic recognition is poor, and the ability to respond to operational information, vehicle status, and open-ended questions is limited. Furthermore, existing human-machine interaction functions are usually relatively independent of vehicle driving control. When a pedestrian approaches the vehicle or makes an inquiry, the vehicle's interaction status may not be coordinated in a timely manner with control processes such as deceleration, stopping, or resuming driving, potentially affecting the safety and continuity of the human-vehicle interaction process. Some interaction devices also require manual activation, configuration, or maintenance, making them unsuitable for the long-term unattended operation requirements of unmanned sanitation vehicles.

[0004] Therefore, in the process of human-machine interaction outside the vehicle of unmanned sanitation vehicles, the difficulty in accurately recognizing pedestrian interaction intentions, the poor stability of voice interaction in complex working environments, and the insufficient coordination between human-machine interaction and vehicle safety control have become problems that urgently need to be solved. Summary of the Invention

[0005] This application provides a multimodal intelligent human-machine interaction control method and system for unmanned sanitation vehicles, aiming to solve the problems in the existing technology of unmanned sanitation vehicles, such as difficulty in accurately recognizing pedestrian interaction intentions, poor stability of voice interaction in complex working environments, and insufficient coordination between human-machine interaction and vehicle safety control.

[0006] In a first aspect, this application provides a multimodal intelligent human-machine interaction control method for an unmanned sanitation vehicle, the method comprising: Continuously acquire visual data, voice data, and positioning data around the unmanned sanitation vehicle, as well as the vehicle status data of the unmanned sanitation vehicle; Perform interactive intent recognition on the visual data and / or the voice data to obtain an interactive intent determination result; When the interaction intent determination result meets the preset activation conditions, a safety control request is sent to the intelligent driving control system of the unmanned sanitation vehicle so that the intelligent driving control system controls the unmanned sanitation vehicle to enter the interactive parking state. In the interactive parking state, the voice data is subjected to voice enhancement processing and voice recognition to obtain interactive text; The interactive text is matched with the local sanitation knowledge base to obtain a local matching result. When the local matching result indicates a successful match, a local parsing result is obtained based on the knowledge base information that matches the interactive text. When the local matching result indicates a failed match, the interactive text and the location data are sent to the cloud processing module for semantic parsing to obtain a cloud parsing result. Based on the local parsing result or the cloud parsing result, and in conjunction with the vehicle status data, generate and output the response content; When the preset interaction end condition is met, a recovery control request is sent to the intelligent driving control system so that the unmanned sanitation vehicle can return from the interactive parking state to the preset working state.

[0007] Optionally, in the above scheme, interaction intent recognition is performed on the visual data and / or the voice data to obtain an interaction intent determination result, including: Pedestrian detection is performed on the visual data to obtain pedestrian detection results, and the pedestrians to be analyzed in the pedestrian detection results are identified as target pedestrians; Face detection and human pose estimation are performed on the target pedestrian to obtain the face region and human key points of the target pedestrian; Based on the face region, facial orientation and gaze direction are estimated, and based on the human body key points, the attention area of ​​the target pedestrian is determined to obtain visual interaction features; The voice data is subjected to voice wake-up detection and keyword recognition to obtain voice interaction features; Based on the visual interaction features and / or the voice interaction features, the interaction intention of the target pedestrian relative to the unmanned sanitation vehicle is determined, and the interaction intention determination result is obtained.

[0008] Optionally, in the above scheme, when the interaction intent determination result meets the preset activation condition, a safety control request is sent to the intelligent driving control system of the unmanned sanitation vehicle, so that the intelligent driving control system controls the unmanned sanitation vehicle to enter an interactive parking state, including: A temporal analysis is performed on the continuously acquired visual interaction features to determine the duration for which the target pedestrian's facial orientation and gaze direction meet the preset attention conditions. Based on the duration and / or the voice interaction features, determine whether the target pedestrian meets the preset activation conditions and obtain the interaction activation determination result; When the interaction activation judgment result indicates that the preset activation condition is met, the current operating status of the unmanned sanitation vehicle is obtained; The safety control request is generated based on the current operating status, and the safety control request is sent to the intelligent driving control system; The intelligent driving control system controls the vehicle controller and the brake-by-wire module to perform vehicle parking operations, enabling the unmanned sanitation vehicle to enter the interactive parking state. Based on the interactive parking status, an interactive prompt message is generated and output through the prompt light group and / or the vehicle speaker.

[0009] Optionally, in the above scheme, during the interactive parking state, the voice data undergoes voice enhancement processing and voice recognition to obtain interactive text, including: Beamforming processing is performed on the voice data acquired by the microphone array to obtain directional voice data; The directional speech data is subjected to noise suppression, echo cancellation, and invalid speech frame removal to obtain enhanced speech data; The enhanced speech data is subjected to speech activity detection to obtain a target speech segment containing valid speech content; The target speech segment is subjected to speech recognition and text conversion to obtain the interactive text.

[0010] Optionally, in the above scheme, the interactive text is matched with a local sanitation knowledge base to obtain a local matching result, including: The interactive text is subjected to keyword extraction and intent classification to obtain interactive keywords and interactive intent categories; Construct local search criteria based on the interactive keywords and the interactive intent categories; Based on the local search criteria, at least one of the following—sanitation operation specifications, vehicle parameter information, and frequently asked questions and answers—is retrieved from the local sanitation knowledge base to obtain the local matching result. When the local matching result indicates a successful match, the local parsing result is generated based on the knowledge base entries that match the local search conditions. When the local matching result indicates a matching failure, cloud parsing trigger information is generated.

[0011] Optionally, in the above scheme, when the local matching result indicates a matching failure, the interactive text and the location data are sent to the cloud processing module for semantic parsing to obtain the cloud parsing result, including: In response to the cloud-based parsing trigger information, the interactive text is combined with the location data to obtain cloud-based search prompt information; The cloud search prompt information is sent to the cloud processing module, so that the cloud processing module performs semantic understanding and search enhancement on the cloud search prompt information to obtain the cloud semantic processing result. Extract at least one of text information, domain category, intent tag and entity slot from the cloud semantic processing results to obtain interactive semantic information; The cloud-based parsing result is generated based on the interactive semantic information.

[0012] Optionally, in the above scheme, the output of the response content includes: The response content is processed by speech synthesis to obtain the response audio signal; The response audio signal is sent to the vehicle speaker so that the response content can be played through the vehicle speaker; The interactive text and the response content are converted into display information, and the display information is sent to the human-computer interaction display screen for synchronous display. Based on the current interactive state of the unmanned sanitation vehicle, light prompt information is generated, and the prompt light group is controlled to output light signals corresponding to the current interactive state.

[0013] Optionally, in the above scheme, before continuously acquiring the visual data, the voice data, and the positioning data, the following steps are also included: The system acquires the vehicle power-on signal of the unmanned sanitation vehicle and activates the edge computing unit and the intelligent interactive service in the edge computing unit based on the vehicle power-on signal. The edge computing unit establishes a local vehicle communication connection and a cloud communication connection to obtain the communication connection status. Based on the communication connection status, an in-vehicle communication connection is established between the edge computing unit and the intelligent driving control system, and the vehicle control handshake result between the intelligent driving control system and the vehicle control execution module is obtained; When the vehicle communication connection is successfully established and the vehicle control handshake result indicates that the handshake was successful, system startup completion information is generated; Based on the system startup completion information, multimodal data acquisition is initiated, and the human-machine interaction state of the unmanned sanitation vehicle is put into standby mode.

[0014] Optionally, in the above scheme, when a preset interaction end condition is met, a recovery control request is sent to the intelligent driving control system to restore the unmanned sanitation vehicle from the interactive parking state to the preset working state, including: After outputting the response content, the effective voice input status and the pedestrian presence status within the preset interaction area are continuously detected to obtain the interaction continuity status; When the interaction duration is characterized by no valid voice input detected within a preset waiting time, or when there are no pedestrians in the preset interaction area, an interaction termination determination result is generated. The recovery control request is generated based on the interaction termination determination result, and the recovery control request is sent to the intelligent driving control system; The intelligent driving control system controls the vehicle control execution module to perform a vehicle recovery operation, so that the unmanned sanitation vehicle is restored to the preset working state, and the work recovery result is obtained. Based on the operation recovery results, the human-machine interaction state of the unmanned sanitation vehicle is switched to standby state.

[0015] Secondly, this application provides a multimodal intelligent human-machine interaction control system for an unmanned sanitation vehicle. The system includes a multimodal perception module, an edge computing unit, an audio-visual output unit, an intelligent driving control system, a vehicle control execution module, a vehicle network module, and a cloud processing module. The edge computing unit is equipped with a local sanitation knowledge base. The multimodal perception module and the acoustic-optical output unit are respectively connected to the edge computing unit, the edge computing unit and the intelligent driving control system are respectively connected to the vehicle network module, the intelligent driving control system is connected to the vehicle control execution module, and the vehicle network module is communicatively connected to the cloud processing module. The multimodal perception module is used to continuously collect visual data, voice data and positioning data around the unmanned sanitation vehicle; The edge computing unit is used to acquire vehicle status data of the unmanned sanitation vehicle, perform interactive intent recognition on the visual data and / or the voice data, obtain interactive intent determination results, and send a safety control request to the intelligent driving control system when the interactive intent determination results meet the preset activation conditions. The intelligent driving control system is used to control the vehicle control execution module to perform a vehicle parking operation according to the safety control request, so that the unmanned sanitation vehicle enters an interactive parking state. The edge computing unit is also used to perform voice enhancement processing and voice recognition on the voice data in the interactive parking state to obtain interactive text, match the interactive text with the local sanitation knowledge base to obtain a local matching result, and obtain a local parsing result when the local matching result indicates a successful match, and send the interactive text and the location data to the cloud processing module through the vehicle network module when the local matching result indicates a failed match. The cloud processing module is used to perform semantic parsing on the interactive text and the location data to obtain cloud parsing results, and then send the cloud parsing results to the edge computing unit through the vehicle network module. The edge computing unit is also used to generate response content based on the local parsing result or the cloud parsing result and in combination with the vehicle status data, send the response content to the audio-visual output unit, and send a recovery control request to the intelligent driving control system when the preset interaction end condition is met. The audio-visual output unit is used to output the response content; The intelligent driving control system is also used to control the vehicle control execution module to perform vehicle recovery operation according to the recovery control request, so that the unmanned sanitation vehicle can be restored from the interactive parking state to the preset working state.

[0016] Compared with the prior art, this application has at least the following beneficial effects: Based on further analysis and research of existing technical problems, this application recognizes the difficulties in accurately identifying pedestrian interaction intentions, poor stability of voice interaction in complex operating environments, and insufficient coordination between human-machine interaction and vehicle safety control during the human-machine interaction process of unmanned sanitation vehicles. By continuously acquiring visual, voice, and positioning data from the surroundings of the unmanned sanitation vehicle, and performing interaction intention recognition on the visual and / or voice data, the vehicle can determine the interaction intention based on the actual behavior and voice information of pedestrians. This improves the accuracy of identifying pedestrian interaction needs outside the vehicle and reduces missed responses and false triggers caused by relying solely on fixed broadcasts or single wake-up methods. When the interaction intention determination result meets preset activation conditions, a safety control request is sent to the intelligent driving control system, causing the vehicle to first enter an interactive parking state, and then perform voice enhancement processing and voice recognition on the voice data, thereby reducing vehicle operating noise and external environmental noise. This paper addresses the impact of noise on speech recognition and establishes a linkage between the human-computer interaction process and the vehicle safety control process. It matches the interactive text with a local sanitation knowledge base, and when local matching fails, uses the interactive text and location data for cloud-based semantic analysis. This ensures that both sanitation professional information and open domain information receive corresponding analysis results. Combined with vehicle status data, it generates and outputs response content that matches the vehicle's current situation, thereby improving the relevance and effectiveness of the interactive response. When the preset interaction termination conditions are met, the vehicle is controlled to return to the preset operating state, allowing the vehicle to continue its original operation after completing the interaction. Therefore, this application forms a complete control closed loop encompassing interactive intent recognition, safe vehicle parking, speech understanding, collaborative response, and operation resumption. This solves the problems of existing unmanned sanitation vehicles, such as difficulty in accurately recognizing pedestrian interaction intent, poor stability of speech interaction in complex operating environments, and insufficient coordination between human-computer interaction and vehicle safety control. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multimodal intelligent human-machine interaction control method for an unmanned sanitation vehicle provided in one embodiment of this application. Figure 2 This is a block diagram of the overall structure of a multimodal intelligent human-machine interaction control system for an unmanned sanitation vehicle provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In one embodiment, such as Figure 1As shown, a multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles is provided. During the unmanned sanitation vehicle's operation of sweeping, cleaning, or garbage transfer, the multimodal perception module continuously acquires visual data, voice data, and positioning data of the surrounding area of ​​the unmanned sanitation vehicle, and the edge computing unit acquires the vehicle status data of the unmanned sanitation vehicle through the vehicle network module.

[0020] The multimodal perception module may include a wide dynamic range camera, a microphone array, and a positioning module. The wide dynamic range camera is mounted on the exterior of the unmanned sanitation vehicle to continuously acquire images of the surrounding environment and send these images as visual data to the edge computing unit. The wide dynamic range camera can be a wide dynamic range starlight-level camera to adapt to sanitation operation environments with strong light, shadows, insufficient nighttime lighting, or significant changes in brightness.

[0021] The microphone array is positioned on the outside of the driverless sanitation vehicle to collect sound signals from the surrounding environment and send these signals as voice data to the edge computing unit. The microphone array can be a circular six-microphone array or a linear four-microphone array, and can be installed at a height appropriate to the voice input position of pedestrians outside the vehicle to improve the sound acquisition effect.

[0022] The positioning module is used to obtain the current location of the unmanned sanitation vehicle and generate positioning data. The positioning data can be used to determine the location where the interaction occurs. When no corresponding knowledge content is found in the local sanitation knowledge base, it can also be sent to the cloud processing module along with the interaction text to support queries related to regional information such as weather, geographic location, and public services.

[0023] Vehicle status data can be provided by the intelligent driving control system, vehicle controller, or vehicle task-related modules, and transmitted to the edge computing unit via the vehicle network module. Vehicle status data may include the vehicle's current driving status, parking status, or sanitation operation status, as well as at least one of the following: vehicle parameters, current task, task progress, and estimated remaining operation time.

[0024] The edge computing unit performs interaction intent recognition on visual and / or voice data to obtain interaction intent determination results. These results characterize whether pedestrians around the vehicle intend to interact with the autonomous sanitation vehicle. The edge computing unit can determine whether a pedestrian is continuously paying attention to the autonomous sanitation vehicle based on their facial orientation, gaze direction, and posture; it can also determine whether a pedestrian has initiated an interaction based on the presence of wake-up phrases, inquiries, or requests for help in the voice data.

[0025] When the interaction intent determination result meets the preset activation conditions, the edge computing unit sends a safety control request to the intelligent driving control system through the vehicle network module. After receiving the safety control request, the intelligent driving control system controls the vehicle controller and the brake-by-wire module to perform a vehicle parking operation through the communication link with the vehicle control execution module, so that the unmanned sanitation vehicle enters the interactive parking state.

[0026] In one specific implementation, the edge computing unit communicates with the intelligent driving control system via an in-vehicle Ethernet network, while the intelligent driving control system communicates with the vehicle control execution module via a CAN network. The intelligent driving control system sends corresponding control information to the vehicle controller and the brake-by-wire module via the CAN network. The vehicle controller adjusts the vehicle's drive output, and the brake-by-wire module generates braking force to bring the unmanned sanitation vehicle to a stop.

[0027] After the driverless sanitation vehicle enters the interactive parking state, the edge computing unit controls the indicator lights and / or the on-board speakers to output interactive prompt information to inform pedestrians that the vehicle has recognized the interaction request and entered the interactive state.

[0028] In the interactive parking state, the microphone array continues to collect pedestrian speech. The edge computing unit performs speech enhancement processing and speech recognition on the speech data collected by the microphone array to obtain interactive text. Speech enhancement processing may include beamforming, noise suppression, echo cancellation, invalid speech frame removal, and speech activity detection to extract effective speech content from vehicle operation noise and road environment noise.

[0029] The edge computing unit matches the interactive text with the local sanitation knowledge base to obtain the local matching results. The local sanitation knowledge base can store at least one of the following: sanitation operation specifications, vehicle parameter information, vehicle function descriptions, operation route information, safety regulations, and frequently asked questions and answers.

[0030] When the local matching result indicates a successful match, the edge computing unit obtains the local parsing result based on the knowledge base information matching the interactive text. When the local matching result indicates a failed match, the edge computing unit sends the interactive text and location data to the cloud processing module through the vehicle network module. The cloud processing module performs semantic parsing on the interactive text and location data to obtain the cloud parsing result.

[0031] The edge computing unit generates a response based on local or cloud-based parsing results, combined with vehicle status data. For example, when a pedestrian asks how much longer the current cleaning task will take to complete, the edge computing unit can determine the interaction intent as a task time query based on local or cloud-based parsing results, and generate a corresponding response based on the task progress or estimated remaining task time in the vehicle status data.

[0032] The edge computing unit converts the response content into an audio signal using speech synthesis and plays the response through the vehicle's speakers. Simultaneously, the edge computing unit sends the interactive text and response content to the human-machine interface display screen for display and controls the indicator lights to output corresponding light signals based on the current interactive status of the autonomous sanitation vehicle.

[0033] After the response is output, the edge computing unit continuously detects whether there is any new valid voice input and whether there are still pedestrians in the preset interaction area. When no new valid voice input is detected within the preset waiting time, or when there are no pedestrians in the preset interaction area, the edge computing unit determines that the preset interaction termination condition has been met and sends a resumption control request to the intelligent driving control system.

[0034] The preset interaction area can be set as the area around the autonomous sanitation vehicle where pedestrian images can be captured by a wide dynamic range camera and pedestrian voice can be captured by a microphone array. Its specific range can be preset based on the camera's field of view, the microphone array's pickup range, and the vehicle's dimensions. The preset waiting time can be set based on the response time required for continuous dialogue.

[0035] The intelligent driving control system, based on the recovery control request, controls the vehicle control execution module to perform a vehicle recovery operation, restoring the unmanned sanitation vehicle from an interactive parking state to a preset operating state. The preset operating state can be either the driving state before entering the interactive parking state or the sanitation operation state. After completing the vehicle recovery operation, the intelligent driving control system feeds back the operation recovery result to the edge computing unit.

[0036] By utilizing visual and voice data to identify pedestrians' interaction intentions, the probability of misjudging ordinary passing behavior as interactive behavior can be reduced. Controlling the vehicle to enter an interactive parking state after confirming the interaction intention allows for coordination between the human-computer interaction process and the vehicle safety control process. Hierarchical processing of interactive text through a local sanitation knowledge base and a cloud processing module can balance the response speed for sanitation-related professional issues with the knowledge coverage for open-domain issues. Automatically restoring the preset operating state after the interaction completes forms a complete control closed loop from proactive perception, parking interaction, intelligent response to resumption of operation.

[0037] In one possible embodiment, the vision processing module in the edge computing unit performs pedestrian detection on the visual data acquired by the wide dynamic range camera to obtain pedestrian detection results. The pedestrian detection results may include the detected pedestrians and the corresponding location regions of the pedestrians in the image.

[0038] The visual processing module can identify pedestrians within a preset interaction area from the pedestrian detection results as target pedestrians. When there is one pedestrian in the preset interaction area, that pedestrian is identified as the target pedestrian; when there are multiple pedestrians in the preset interaction area, each pedestrian can be identified as the corresponding target pedestrian, and subsequent interaction intent recognition can be performed separately.

[0039] The vision processing module performs face detection and human pose estimation on the target pedestrian, obtaining the target pedestrian's face region and human key points. Among them, the face region is used to estimate facial orientation and gaze direction, while the human key points are used to characterize the positional relationships of the target pedestrian's head, shoulders, arms, and torso.

[0040] The visual processing module determines the facial orientation and gaze direction of the target pedestrian based on the facial region, and combines this with key points on the human body to determine the pedestrian's body orientation and area of ​​interest, thus obtaining visual interaction features. These visual interaction features may include at least one of the following: whether the target pedestrian's face is facing the driverless sanitation vehicle, whether their gaze is directed at the driverless sanitation vehicle, whether their body is facing the driverless sanitation vehicle, and whether they are displaying an interactive gesture such as raising a hand.

[0041] The speech processing module in the edge computing unit performs speech wake-up detection and keyword recognition on the speech data collected by the microphone array to obtain speech interaction features. These features may include at least one of the following: whether valid speech was detected, whether a preset wake-up phrase was detected, whether an inquiry phrase was detected, or whether a request for help was detected.

[0042] The edge computing unit determines the interaction intent of the target pedestrian relative to the driverless sanitation vehicle based on visual interaction features and / or voice interaction features, and obtains the interaction intent determination result. When the target pedestrian's face orientation and gaze direction are continuously pointed at the driverless sanitation vehicle, it can be determined that the target pedestrian has an interaction intent; when voice content that can represent asking for help, requesting assistance, or initiating a dialogue is detected, it can also be determined that the target pedestrian has an interaction intent.

[0043] By using the above implementation methods, the behavior of target pedestrians can be analyzed by utilizing facial regions, key points of the human body, facial orientation, gaze direction, and voice interaction features. This can distinguish between pedestrians who are simply near vehicles and pedestrians who actively face vehicles to initiate communication, thereby improving the accuracy of interaction intent determination results and reducing false activation of human-computer interaction modes.

[0044] In one possible embodiment, the edge computing unit performs temporal analysis on the continuously acquired visual interaction features to determine the duration for which the facial orientation and gaze direction of the target pedestrian meet the preset attention conditions.

[0045] The preset attention conditions may include at least one of the following: the target pedestrian's face is facing the driverless sanitation vehicle, their gaze is directed at the driverless sanitation vehicle, or their body is facing the driverless sanitation vehicle. The edge computing unit continuously determines whether the target pedestrian meets the preset attention conditions according to the acquisition sequence of the visual data, and determines the duration for which the preset attention conditions are met.

[0046] In one specific implementation, the preset duration threshold can be set to one second. When the duration for which the target pedestrian's facial orientation and gaze direction meet the preset attention conditions reaches the preset duration threshold, the target pedestrian is determined to have met the preset activation conditions. In other implementations, corresponding preset duration thresholds can also be set according to the vehicle's operating environment and the requirements for controlling accidental triggering of interactions.

[0047] The edge computing unit can also combine voice interaction features to determine whether a target pedestrian meets preset activation conditions. When the voice interaction feature representation detects inquiries, requests for help, or dialogue initiation, it can generate an interaction activation judgment result indicating that the preset activation conditions are met.

[0048] When the interaction activation judgment result indicates that the preset activation conditions are met, the edge computing unit obtains the current operating status of the unmanned sanitation vehicle through the vehicle network module. The current operating status may include the vehicle being in motion, the vehicle being parked, or the vehicle being performing mobile sanitation operations.

[0049] The edge computing unit generates safety control requests based on the current operating status. When the unmanned sanitation vehicle is driving or performing mobile sanitation operations, the safety control request is used to request control to stop the vehicle; when the unmanned sanitation vehicle is already parked, the safety control request can be used to request that the vehicle remain parked.

[0050] The edge computing unit sends safety control requests to the intelligent driving control system. The intelligent driving control system then sends parking control information to the vehicle controller and brake-by-wire module via the CAN network. The vehicle controller adjusts the vehicle's power output, and the brake-by-wire module generates braking force, causing the unmanned sanitation vehicle to enter an interactive parking state.

[0051] The edge computing unit generates interactive prompts based on the parking status and outputs these prompts via indicator lights and / or in-vehicle speakers. These prompts can indicate that the vehicle is parked, has entered an interactive state, or that the system is receiving pedestrian voice input.

[0052] Through the above implementation method, the interaction intent determination result is linked with the intelligent driving control system, vehicle controller and drive-by-wire braking module to form a control linkage, so that the unmanned sanitation vehicle enters the interactive parking state before carrying out human-machine interaction outside the vehicle. This can avoid continuous dialogue with pedestrians at close range when the vehicle is in motion, and use prompt sounds and light signals to provide feedback on the vehicle status to pedestrians, thereby improving the safety and status perception of the human-machine interaction process.

[0053] In one possible embodiment, the microphone array is a circular six-microphone array or a linear four-microphone array, and is installed on the outside of the unmanned sanitation vehicle. The microphone array collects voice information from the vehicle's surroundings, and the voice processing module in the edge computing unit uses the beamforming function of the microphone array to perform directional sound pickup, obtaining directional voice data.

[0054] The voice processing module performs noise suppression on the directional voice data to reduce the impact of vehicle operation noise, road ambient noise, and other non-target sounds on pedestrian speech. The module also performs echo cancellation on the directional voice data to reduce the impact of prompts or responses played by the vehicle speakers being re-acquired by the microphone array.

[0055] In one optional implementation, the speech processing module includes an acoustic echo cancellation unit (AEC), which is deployed as a software algorithm module within the edge computing unit and is communicatively connected to the microphone array, the speech synthesis module, and the state management module, respectively. The AEC receives a reference signal output from the speech synthesis module and uses the reference signal to perform adaptive echo cancellation processing on the speech data acquired by the microphone array.

[0056] Specifically, when the speech synthesis module generates a response audio signal for playing reply content or interactive prompts, it synchronously sends the audio data corresponding to the response audio signal as a reference signal to the echo cancellation unit. The speech data collected by the microphone array may include pedestrian speech outside the vehicle, environmental noise, and echoes formed by the acoustic propagation of content played by the vehicle speakers. The echo cancellation unit performs adaptive filtering based on the reference signal to determine the echo path estimation result between the vehicle speakers and the microphone array, generates an echo estimation signal based on the echo path estimation result, and filters out the echo estimation signal from the noise-suppressed directional speech data to obtain echo-cancelled speech data.

[0057] The speech processing module removes invalid speech frames from the speech data that has undergone noise suppression and echo cancellation to obtain enhanced speech data. Invalid speech frames may include silent frames that do not contain valid speech content or speech frames that are mainly composed of environmental noise.

[0058] The speech processing module performs speech activity detection on the enhanced speech data, extracts target speech segments containing valid speech content from the enhanced speech data, and performs speech recognition and text conversion on the target speech segments to obtain interactive text. The above speech processing process is combined with the microphone array's beamforming, noise reduction, echo cancellation, and invalid frame removal to adapt to the outdoor operating environment of the unmanned sanitation vehicle.

[0059] Interactive text can be simultaneously sent to the human-computer interaction display screen so that pedestrians or maintenance personnel can view the speech recognition results and confirm whether the input content has been correctly recognized.

[0060] Through the above implementation method, beamforming directional sound pickup is performed using a microphone array, and noise suppression, echo cancellation, invalid speech frame removal, and speech activity detection are executed sequentially. This reduces the interference of vehicle operation noise and road environment noise on pedestrian speech. At the same time, by using the response audio signal output by the speech synthesis module as the reference signal for echo cancellation processing, the echo components formed by the content played by the vehicle speaker in the microphone array can be estimated and filtered out. This avoids recognizing the response content or interactive prompts output by the vehicle itself as the speech input of pedestrians outside the vehicle, thereby preventing the system from having repeated responses or self-answering, and improving the accuracy of speech recognition and the stability of continuous interaction in complex outdoor acoustic environments.

[0061] In one possible embodiment, the edge computing unit performs keyword extraction and intent classification on the interactive text to obtain interactive keywords and interactive intent categories.

[0062] Interactive keywords are used to characterize the question or query content in the interactive text. Interactive intent categories may include at least one of the following: sanitation operation information query, vehicle function query, route information query, safety regulations query, vehicle status query, request for help, and open domain information query.

[0063] The edge computing unit constructs local search conditions based on interaction keywords and interaction intent categories, and retrieves corresponding knowledge content from the local sanitation knowledge base based on the local search conditions.

[0064] The local sanitation knowledge base is located in the local storage space of the edge computing unit and can store at least one of the following: sanitation operation specifications, vehicle parameter information, vehicle function descriptions, operation route information, safety regulations, vehicle task information, and frequently asked questions and answers.

[0065] Edge computing units can determine local matching results based on the degree of matching between interaction keywords, interaction intent categories, and knowledge base entries. When the corresponding matching degree meets the preset matching conditions, the local matching result indicates a successful match; when the corresponding matching degree does not meet the preset matching conditions, or no corresponding knowledge base entry is retrieved, the local matching result indicates a failed match.

[0066] Local matching results can include a successful match status, a failed match status, and the corresponding knowledge base entry information when a match is successful.

[0067] When the local matching result indicates a successful match, the edge computing unit extracts the standard response content, parameter fields, or data call identifier from the matching knowledge base entry and generates a local parsing result.

[0068] For vehicle status queries or task progress queries, the local parsing results can include a data call identifier that instructs the edge computing unit to read the corresponding real-time information from the vehicle status data, so that the response content can be generated in combination with the actual vehicle status.

[0069] When the local matching result indicates a matching failure, the edge computing unit generates cloud parsing trigger information to initiate the combination of interactive text and location data and the cloud transmission process.

[0070] By prioritizing the use of sanitation operation specifications, vehicle parameters, and frequently asked questions and answers stored locally on the vehicle to handle sanitation-related issues, the reliance on cloud communication links can be reduced and the response speed for common questions can be improved. Cloud parsing is triggered only when the local sanitation knowledge base fails to match, which can expand the range of issues that the system can handle while maintaining the stability of professional knowledge responses.

[0071] In one possible embodiment, the edge computing unit responds to the cloud-based parsing trigger information by combining the interactive text with the location data corresponding to the current location of the unmanned sanitation vehicle to obtain cloud-based retrieval prompt information.

[0072] For issues involving weather, geography, or public services, location data can be used to limit the geographical scope of cloud-based knowledge retrieval and content generation.

[0073] The edge computing unit sends cloud-based search prompts to the cloud processing module via the vehicle network module and 4G / 5G communication links. The cloud processing module may include a large cloud model, a knowledge interface, a search enhancement generation unit, and a vehicle communication module.

[0074] The cloud-based large-scale model can employ a Transformer architecture for semantic understanding and intent parsing of cloud-based search prompts. The knowledge interface is used to retrieve general knowledge, weather information, geographic information, or public service information based on the semantic understanding results. The search enhancement generation unit generates cloud-based semantic processing results based on the retrieved knowledge content.

[0075] The cloud processing module determines at least one of text information, domain category, intent label and entity slot from the cloud semantic processing results to obtain interactive semantic information.

[0076] Among them, text information can represent the candidate response content generated in the cloud; domain category can represent the knowledge domain to which the interactive text belongs; intent tag can represent the pedestrian's query, request for help or dialogue intent; entity slot can represent the location, time, vehicle task or query object involved in the interactive text.

[0077] The cloud processing module generates cloud parsing results based on the interactive semantic information, and returns the cloud parsing results to the edge computing unit through the vehicle network module.

[0078] The cloud processing module is primarily used for processing general knowledge and open-domain information in human-computer interaction. The intelligent driving cloud platform or remote control platform can serve as an independent cloud-based vehicle management platform, used to monitor the driving status data of unmanned sanitation vehicles, implement remote control, and perform online upgrades. Both the cloud processing module and the intelligent driving cloud platform or remote control platform can communicate with the vehicle network module via 4G / 5G communication links.

[0079] Through the above implementation method, the interactive text and location data are combined and sent to the cloud processing module. The cloud-based large model and retrieval enhancement generation unit are used to perform semantic understanding and knowledge retrieval. This can generate cloud-based analysis results for questions not covered by the local sanitation knowledge base, such as weather, geography, and public services, thereby expanding the range of interactive content that unmanned sanitation vehicles can understand and respond to.

[0080] In one possible embodiment, the speech synthesis module in the edge computing unit receives the reply content and converts the text-based reply content into audio format to obtain the reply audio signal.

[0081] The response audio signal can be sent to an audio amplifier, which amplifies the signal and then drives one or more vehicle-mounted speakers located on the exterior of the vehicle to play the response content. Multiple vehicle-mounted speakers can be arranged according to the vehicle's structure to ensure that pedestrians in different directions receive the response content.

[0082] While the vehicle's speakers play the response content, the speech synthesis module also simultaneously sends the audio data corresponding to the response audio signal to the echo cancellation unit as a reference signal for adaptive echo cancellation processing. Thus, the response audio signal forms a playback branch and a reference branch. The playback branch is sent to the vehicle's speakers via an audio amplifier, while the reference branch is sent to the echo cancellation unit. The echo cancellation unit filters out the speaker echo components from the speech data collected by the microphone array based on the reference signal in the reference branch.

[0083] The edge computing unit converts interactive text and response content into display information and sends the display information to the human-computer interaction display screen for synchronous display.

[0084] The human-machine interface display can connect to the edge computing unit via HDMI and Universal Serial Bus interfaces, and supports touch input and mouse input. By displaying interactive text, pedestrians or maintenance personnel can view the voice content recognized by the system; by displaying response content, it can provide interactive information in text form when the external environment is noisy or pedestrians cannot clearly hear the voice.

[0085] The human-machine interface display screen can also be used to display the configuration interface, allowing maintenance personnel to input or adjust relevant configuration parameters via touch. Indicator lights can be distributed on the exterior of the driverless sanitation vehicle and output corresponding light signals when the vehicle is in different interactive states.

[0086] The edge computing unit generates lighting prompts based on the current interaction state of the autonomous sanitation vehicle and controls the prompt light group to output light signals corresponding to the current interaction state. The current interaction state can include interaction activation state, interaction parking state, voice reception state, response output state, and resumption of work state.

[0087] The state management module can also determine the voice playback stage information based on the current interaction state of the unmanned sanitation vehicle and send the voice playback stage information to the echo cancellation unit. The voice playback stage information can be used to characterize the voice playback stage (not yet played, voice playback begins, voice playback continues, or voice playback ends). The echo cancellation unit adjusts adaptive filtering parameters based on the voice playback stage information to adapt the echo cancellation process to the voice playback process of the vehicle's speakers. The adaptive filtering parameters may include at least one of filter weight update parameters, convergence step size, and echo path tracking parameters to improve echo tracking capability when the vehicle's voice playback state changes.

[0088] By implementing the above methods, the response content is converted into voice output and text display, and combined with the vehicle interaction status output by the indicator light group, pedestrians can obtain vehicle feedback through auditory and visual means, and can confirm whether the system accurately recognizes the input content by displaying interactive text, thereby improving the clarity and user-friendliness of human-computer interaction outside the vehicle.

[0089] In one possible embodiment, the edge computing unit is connected to the vehicle power supply system of the autonomous sanitation vehicle. When the vehicle is powered on, the vehicle power supply system provides a vehicle power-on signal to the edge computing unit.

[0090] The edge computing unit automatically starts the operating system and loads intelligent interactive services based on the vehicle's power-on signal. These intelligent interactive services may include visual processing services, voice processing services, voice synthesis services, local sanitation knowledge base services, and interactive status management services.

[0091] The edge computing unit can utilize the RK3588 platform and run a Linux operating system. It supports Type-C, Universal Serial Bus, HDMI, and Ethernet interfaces, and features power-on startup, AI inference, local knowledge base storage, and Socket communication. While the RK3588 platform represents one specific hardware implementation, other edge computing devices with corresponding processing capabilities and communication interfaces can also be used.

[0092] The edge computing unit initiates a Socket communication service and establishes a local communication connection with the vehicle network module via the vehicle's Ethernet. The vehicle network module may include an industrial switch, a vehicle gateway, and a 4G / 5G communication module.

[0093] The edge computing unit establishes a cloud communication connection with the cloud processing module through the vehicle network module. The in-vehicle Ethernet can serve as the main communication link within the vehicle, while the 4G / 5G communication link is used to connect the cloud processing module and the intelligent driving cloud platform or remote control platform.

[0094] After the local vehicle communication connection is established, the edge computing unit establishes an in-vehicle communication connection with the intelligent driving control system. The intelligent driving control system then performs a vehicle control handshake with the vehicle control execution module via the CAN network. The edge computing unit then obtains the handshake result between the intelligent driving control system and the vehicle control execution module.

[0095] When the vehicle communication connection between the edge computing unit and the intelligent driving control system is successfully established, and the vehicle control handshake result indicates that the handshake was successful, the edge computing unit generates system startup completion information.

[0096] The edge computing unit activates the wide dynamic range camera, microphone array, and positioning module based on the system startup completion information, and then puts the human-machine interaction of the unmanned sanitation vehicle into standby mode through the state management module. In standby mode, the unmanned sanitation vehicle can perform sanitation operations normally while continuously detecting whether there are any interaction intentions in the vehicle's surroundings.

[0097] The edge computing unit can also receive remote configuration commands via Socket communication service to remotely configure voice pickup sensitivity, pedestrian recognition threshold, parking trigger distance, lighting logic, broadcast volume, and knowledge base response scripts.

[0098] The edge computing unit can also upload equipment operating status, interaction logs and fault codes to the cloud through the vehicle network module, and can receive algorithm models and local sanitation knowledge base update data sent by the cloud to realize online upgrades of algorithm models and local sanitation knowledge base.

[0099] Through the above implementation methods, the edge computing unit, intelligent interactive service and corresponding communication link are automatically started by the vehicle power-on signal, and the system automatically enters standby mode after completing the vehicle control handshake. This enables the human-machine interaction system to adapt to the unmanned operation mode of driverless sanitation vehicles. Through remote configuration, status uploading and online upgrades, the manual operation required for on-site debugging and maintenance can be reduced.

[0100] In one possible embodiment, the edge computing unit keeps the driverless sanitation vehicle in an interactive parking state after the onboard speaker plays the response content.

[0101] The speech processing module in the edge computing unit continuously detects whether there are new valid speech inputs in the speech data collected by the microphone array, and the vision processing module continuously detects whether there are still pedestrians in the preset interaction area in the vision data collected by the wide dynamic range camera, thus obtaining the interaction continuity status.

[0102] When a new valid voice input is detected, the edge computing unit continues to perform voice enhancement, voice recognition, and response processing on the new voice data to support continuous interaction between pedestrians and the driverless sanitation vehicle.

[0103] When the interaction continues and no valid voice input is detected within a preset waiting time, or when there are no pedestrians in the preset interaction area, the edge computing unit generates an interaction termination determination result.

[0104] The edge computing unit generates a recovery control request based on the interaction termination determination result and sends the recovery control request to the intelligent driving control system through the vehicle network module.

[0105] After receiving a recovery control request, the intelligent driving control system, based on the task information prior to entering the interactive parking state, controls the unmanned sanitation vehicle to resume driving or continue performing the cleaning task through the vehicle control execution module, and generates a task recovery result. The intelligent driving control system then feeds the task recovery result back to the edge computing unit.

[0106] When the operation recovery result indicates that the unmanned sanitation vehicle has returned to the preset operation state, the state management module ends the current interaction process and switches the human-machine interaction state of the unmanned sanitation vehicle to standby state in order to continue to detect new interaction intentions.

[0107] For example, when an unmanned sanitation vehicle is performing a cleaning task in the park, a pedestrian faces the vehicle and raises their hand to signal. The wide dynamic range camera captures the pedestrian's image, and the edge computing unit determines the interaction intention based on facial orientation, gaze direction, and human posture. Then, the intelligent driving control system controls the vehicle to enter an interactive parking state.

[0108] When a pedestrian asks, "How much longer will it take to finish cleaning?", the edge computing unit converts the pedestrian's speech into interactive text and determines that the interaction intent is a time query, with the query target being the end time of cleaning. The edge computing unit combines vehicle task-related data to obtain the estimated remaining operation time and generates a response message, "Estimated to take another 25 minutes," which is played through the vehicle's speaker and displayed on the human-machine interface screen.

[0109] When the pedestrian stops inputting new voice or leaves the preset interaction area, the edge computing unit generates a recovery control request, and the intelligent driving control system controls the unmanned sanitation vehicle to exit the interactive parking state and resume cleaning operations.

[0110] Through the above implementation method, after the response content is output, the effective voice input status and pedestrian presence status are continuously detected. The interactive parking status can be maintained and continuous interaction can be supported while the pedestrian continues to ask questions. The original operation status can be restored in time after the pedestrian ends the conversation or leaves, thereby avoiding premature resumption of driving and affecting the interaction, and also avoiding the vehicle from being parked unnecessarily for a long time.

[0111] In one embodiment, such as Figure 2 As shown, the multimodal intelligent human-machine interaction control system of the unmanned sanitation vehicle includes a multimodal perception module, an edge computing unit, an audio-visual output unit, an intelligent driving control system, a vehicle control execution module, a vehicle network module, and a cloud processing module. The edge computing unit is equipped with a local sanitation knowledge base.

[0112] The multimodal perception module and the audio-visual output unit are connected to the edge computing unit, the edge computing unit and the intelligent driving control system are connected to the vehicle network module, the intelligent driving control system is connected to the vehicle control execution module, and the vehicle network module is communicatively connected to the cloud processing module.

[0113] The multimodal perception module includes a wide dynamic range camera, a microphone array, and a positioning module. The wide dynamic range camera is used to continuously collect visual data about the surroundings of the autonomous sanitation vehicle, the microphone array is used to continuously collect voice data about the surroundings of the vehicle, and the positioning module is used to obtain the positioning data of the autonomous sanitation vehicle.

[0114] Edge computing units can be deployed inside the cabin of unmanned sanitation vehicles and connected to multimodal perception modules, audio-visual output units, vehicle network modules, and intelligent driving control systems.

[0115] The edge computing unit can run a self-starting module, a vision processing module, a voice processing module, a local sanitation knowledge base, a voice synthesis module, and a status management module.

[0116] The self-starting module is used to activate the edge computing unit and intelligent interactive services based on the vehicle's power-on signal. The vision processing module is used to perform pedestrian detection, face detection, human pose estimation, facial orientation estimation, and gaze direction estimation on visual data. The speech processing module is used to perform beamforming, noise suppression, echo cancellation, speech activity detection, speech recognition, keyword extraction, and intent classification on speech data.

[0117] The speech processing module may include an echo cancellation unit, which is connected to a microphone array, a speech synthesis module, and a state management module. The speech synthesis module provides the echo cancellation unit with a reference signal corresponding to the returned audio signal. The microphone array provides the echo cancellation unit with speech data including pedestrian voices from outside the vehicle, environmental noise, and speaker echoes. The state management module provides the echo cancellation unit with speech playback stage information. Based on the reference signal and the speech playback stage information, the echo cancellation unit performs adaptive echo cancellation processing on the speech data and sends the processed speech data to the speech activity detection and speech recognition processing stages.

[0118] The local sanitation knowledge base stores sanitation operation specifications, frequently asked questions and answers, vehicle parameter information, vehicle function descriptions, and safety regulations. The speech synthesis module converts responses into audio signals. The status management module manages the switching between standby, interactive active, interactive stopped, response output, and resume operation states.

[0119] The audio-visual output unit may include an audio amplifier, one or more vehicle speakers, a human-machine interface display screen, and an indicator light assembly. The audio amplifier amplifies the response audio signal, the vehicle speakers play the response content and interactive prompts, the human-machine interface display screen displays interactive text, response content, and the configuration interface, and the indicator light assembly outputs light signals corresponding to the vehicle's current interactive state.

[0120] The vehicle network module may include an industrial switch, a vehicle gateway, and a 4G / 5G communication module. The edge computing unit and the intelligent driving control system can access the vehicle network module via in-vehicle Ethernet. The edge computing unit can be connected to the multimodal perception module and the audio-visual output unit via a universal serial bus interface, an HDMI interface, or corresponding hardware interfaces.

[0121] The intelligent driving control system may include a domain controller for vehicle driving control. The vehicle control execution module may include a vehicle controller and a brake-by-wire module. The intelligent driving control system connects to the vehicle control execution module via a CAN network to control the vehicle controller and brake-by-wire module to perform vehicle parking operations based on safety control requests, and to control the vehicle control execution module to perform vehicle recovery operations based on recovery control requests.

[0122] The cloud processing module is deployed on a cloud server and may include a large cloud model, a knowledge interface, a retrieval enhancement generation unit, and a vehicle communication module. The cloud processing module communicates with the edge computing unit via a 4G / 5G communication link and the vehicle network module to receive interactive text and location data, and generate cloud-based parsing results.

[0123] The intelligent driving cloud platform or remote control platform can serve as a cloud-based vehicle management platform, connecting to the vehicle's network module via a 4G / 5G communication link. It is used to monitor vehicle driving status, implement remote control, and perform online upgrades. The cloud processing module primarily handles human-machine interaction information, while the intelligent driving cloud platform or remote control platform mainly handles vehicle management and intelligent driving information.

[0124] During system operation, the multimodal perception module sends visual data, voice data, and positioning data to the edge computing unit. The edge computing unit generates an interaction intent determination result based on the visual data and / or voice data. When the interaction intent determination result meets the preset activation conditions, the edge computing unit sends a safety control request to the intelligent driving control system, which then controls the vehicle control execution module to put the unmanned sanitation vehicle into an interactive parking state.

[0125] The edge computing unit processes the voice data in the interactive parking state to obtain the interactive text, and prioritizes matching it with the local sanitation knowledge base. When the local matching is successful, a local parsing result is generated; when the local matching fails, the interactive text and location data are sent to the cloud processing module through the vehicle network module, which generates a cloud parsing result and returns it to the edge computing unit.

[0126] The edge computing unit generates a response based on local or cloud-based parsing results and vehicle status data, and outputs the response through the audio-visual output unit. When the preset interaction termination conditions are met, the edge computing unit sends a recovery control request to the intelligent driving control system. The intelligent driving control system then controls the vehicle control execution module to restore the unmanned sanitation vehicle to the preset operating state and feeds back the operation recovery result to the edge computing unit.

[0127] The specific division of the above modules can be adjusted according to the hardware architecture of the unmanned sanitation vehicle. For example, the vision processing module and the voice processing module can be set in the same edge computing device, or they can be deployed in different processing devices; the vehicle network module can consist of an independent industrial switch and a vehicle gateway, or it can be integrated into the existing vehicle communication equipment.

[0128] The multimodal interaction intent recognition, safe parking control, collaborative response between the local sanitation knowledge base and the cloud processing module, and the handling method for resuming driving after the interaction can also be applied to other low-speed unmanned operating vehicles, such as unmanned patrol vehicles, unmanned delivery vehicles, or unmanned sweepers.

[0129] Through the above implementation methods, by utilizing the collaborative connection between the multimodal perception module, edge computing unit, intelligent driving control system, vehicle control execution module, audio-visual output unit and cloud processing module, a system link is formed from external information collection, interactive intent recognition, parking control, semantic parsing, response output to recovery operation. This combines the external human-machine interaction function with the driving control of the unmanned sanitation vehicle, and improves the system's real-time performance, knowledge coverage and adaptability to the vehicle environment through the collaboration of local processing and cloud processing.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multimodal intelligent human-machine interaction control method for an unmanned sanitation vehicle, characterized in that, include: Continuously acquire visual data, voice data, and positioning data around the unmanned sanitation vehicle, as well as the vehicle status data of the unmanned sanitation vehicle; Perform interactive intent recognition on the visual data and / or the voice data to obtain an interactive intent determination result; When the interaction intent determination result meets the preset activation conditions, a safety control request is sent to the intelligent driving control system of the unmanned sanitation vehicle so that the intelligent driving control system controls the unmanned sanitation vehicle to enter the interactive parking state. In the interactive parking state, the voice data is subjected to voice enhancement processing and voice recognition to obtain interactive text; The interactive text is matched with the local sanitation knowledge base to obtain a local matching result. When the local matching result indicates a successful match, a local parsing result is obtained based on the knowledge base information that matches the interactive text. When the local matching result indicates a failed match, the interactive text and the location data are sent to the cloud processing module for semantic parsing to obtain a cloud parsing result. Based on the local parsing result or the cloud parsing result, and in conjunction with the vehicle status data, generate and output the response content; When the preset interaction end condition is met, a recovery control request is sent to the intelligent driving control system so that the unmanned sanitation vehicle can return from the interactive parking state to the preset working state.

2. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 1, characterized in that, Perform interaction intent recognition on the visual data and / or the voice data to obtain an interaction intent determination result, including: Pedestrian detection is performed on the visual data to obtain pedestrian detection results, and the pedestrians to be analyzed in the pedestrian detection results are identified as target pedestrians; Face detection and human pose estimation are performed on the target pedestrian to obtain the face region and human key points of the target pedestrian; Based on the face region, facial orientation and gaze direction are estimated, and based on the human body key points, the attention area of ​​the target pedestrian is determined to obtain visual interaction features; The voice data is subjected to voice wake-up detection and keyword recognition to obtain voice interaction features; Based on the visual interaction features and / or the voice interaction features, the interaction intention of the target pedestrian relative to the unmanned sanitation vehicle is determined, and the interaction intention determination result is obtained.

3. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 2, characterized in that, When the interaction intent determination result meets the preset activation conditions, a safety control request is sent to the intelligent driving control system of the unmanned sanitation vehicle, so that the intelligent driving control system controls the unmanned sanitation vehicle to enter an interactive parking state, including: A temporal analysis is performed on the continuously acquired visual interaction features to determine the duration for which the target pedestrian's facial orientation and gaze direction meet the preset attention conditions. Based on the duration and / or the voice interaction features, determine whether the target pedestrian meets the preset activation conditions and obtain the interaction activation determination result; When the interaction activation judgment result indicates that the preset activation condition is met, the current operating status of the unmanned sanitation vehicle is obtained; The safety control request is generated based on the current operating status, and the safety control request is sent to the intelligent driving control system; The intelligent driving control system controls the vehicle controller and the brake-by-wire module to perform vehicle parking operations, enabling the unmanned sanitation vehicle to enter the interactive parking state. Based on the interactive parking status, an interactive prompt message is generated and output through the prompt light group and / or the vehicle speaker.

4. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 1, characterized in that, In the interactive parking state, the voice data is subjected to voice enhancement processing and voice recognition to obtain interactive text, including: Beamforming processing is performed on the voice data acquired by the microphone array to obtain directional voice data; The directional speech data is subjected to noise suppression, echo cancellation, and invalid speech frame removal to obtain enhanced speech data; The enhanced speech data is subjected to speech activity detection to obtain a target speech segment containing valid speech content; The target speech segment is subjected to speech recognition and text conversion to obtain the interactive text.

5. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 1, characterized in that, The interactive text is matched against the local sanitation knowledge base to obtain local matching results, including: The interactive text is subjected to keyword extraction and intent classification to obtain interactive keywords and interactive intent categories; Construct local search criteria based on the interactive keywords and the interactive intent categories; Based on the local search criteria, at least one of the following—sanitation operation specifications, vehicle parameter information, and frequently asked questions and answers—is retrieved from the local sanitation knowledge base to obtain the local matching result. When the local matching result indicates a successful match, the local parsing result is generated based on the knowledge base entries that match the local search conditions. When the local matching result indicates a matching failure, cloud parsing trigger information is generated.

6. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 5, characterized in that, When the local matching result indicates a matching failure, the interactive text and the location data are sent to the cloud processing module for semantic parsing to obtain the cloud parsing result, including: In response to the cloud-based parsing trigger information, the interactive text is combined with the location data to obtain cloud-based search prompt information; The cloud search prompt information is sent to the cloud processing module, so that the cloud processing module performs semantic understanding and search enhancement on the cloud search prompt information to obtain the cloud semantic processing result. Extract at least one of text information, domain category, intent tag and entity slot from the cloud semantic processing results to obtain interactive semantic information; The cloud-based parsing result is generated based on the interactive semantic information.

7. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 1, characterized in that, The output of the response content includes: The response content is processed by speech synthesis to obtain the response audio signal; The response audio signal is sent to the vehicle speaker so that the response content can be played through the vehicle speaker; The interactive text and the response content are converted into display information, and the display information is sent to the human-computer interaction display screen for synchronous display. Based on the current interactive state of the unmanned sanitation vehicle, light prompt information is generated, and the prompt light group is controlled to output light signals corresponding to the current interactive state.

8. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 1, characterized in that, Before continuously acquiring the visual data, the voice data, and the location data, the method further includes: The system acquires the vehicle power-on signal of the unmanned sanitation vehicle and activates the edge computing unit and the intelligent interactive service in the edge computing unit based on the vehicle power-on signal. The edge computing unit establishes a local vehicle communication connection and a cloud communication connection to obtain the communication connection status. Based on the communication connection status, an in-vehicle communication connection is established between the edge computing unit and the intelligent driving control system, and the vehicle control handshake result between the intelligent driving control system and the vehicle control execution module is obtained; When the vehicle communication connection is successfully established and the vehicle control handshake result indicates that the handshake was successful, system startup completion information is generated; Based on the system startup completion information, multimodal data acquisition is initiated, and the human-machine interaction state of the unmanned sanitation vehicle is put into standby mode.

9. The multimodal intelligent human-machine interaction control method for unmanned sanitation vehicles according to claim 7, characterized in that, When the preset interaction termination condition is met, a recovery control request is sent to the intelligent driving control system to restore the unmanned sanitation vehicle from the interactive parking state to the preset working state, including: After outputting the response content, the effective voice input status and the pedestrian presence status within the preset interaction area are continuously detected to obtain the interaction continuity status; When the interaction duration is characterized by no valid voice input detected within a preset waiting time, or when there are no pedestrians in the preset interaction area, an interaction termination determination result is generated. The recovery control request is generated based on the interaction termination determination result, and the recovery control request is sent to the intelligent driving control system; The intelligent driving control system controls the vehicle control execution module to perform a vehicle recovery operation, so that the unmanned sanitation vehicle is restored to the preset working state, and the work recovery result is obtained. Based on the operation recovery results, the human-machine interaction state of the unmanned sanitation vehicle is switched to standby state.

10. A multimodal intelligent human-machine interaction control system for an unmanned sanitation vehicle, characterized in that, It includes a multimodal perception module, an edge computing unit, an audio-visual output unit, an intelligent driving control system, a vehicle control execution module, a vehicle network module, and a cloud processing module. The edge computing unit is equipped with a local sanitation knowledge base. The multimodal perception module and the acoustic-optical output unit are respectively connected to the edge computing unit, the edge computing unit and the intelligent driving control system are respectively connected to the vehicle network module, the intelligent driving control system is connected to the vehicle control execution module, and the vehicle network module is communicatively connected to the cloud processing module. The multimodal perception module is used to continuously collect visual data, voice data and positioning data around the unmanned sanitation vehicle; The edge computing unit is used to acquire vehicle status data of the unmanned sanitation vehicle, perform interactive intent recognition on the visual data and / or the voice data, obtain interactive intent determination results, and send a safety control request to the intelligent driving control system when the interactive intent determination results meet the preset activation conditions. The intelligent driving control system is used to control the vehicle control execution module to perform a vehicle parking operation according to the safety control request, so that the unmanned sanitation vehicle enters an interactive parking state. The edge computing unit is also used to perform voice enhancement processing and voice recognition on the voice data in the interactive parking state to obtain interactive text, match the interactive text with the local sanitation knowledge base to obtain a local matching result, and obtain a local parsing result when the local matching result indicates a successful match, and send the interactive text and the location data to the cloud processing module through the vehicle network module when the local matching result indicates a failed match. The cloud processing module is used to perform semantic parsing on the interactive text and the location data to obtain cloud parsing results, and then send the cloud parsing results to the edge computing unit through the vehicle network module. The edge computing unit is also used to generate response content based on the local parsing result or the cloud parsing result and in combination with the vehicle status data, send the response content to the audio-visual output unit, and send a recovery control request to the intelligent driving control system when the preset interaction end condition is met. The audio-visual output unit is used to output the response content; The intelligent driving control system is also used to control the vehicle control execution module to perform vehicle recovery operation according to the recovery control request, so that the unmanned sanitation vehicle can be restored from the interactive parking state to the preset working state.

Images