A human-machine interaction system and method based on a service intelligent robot

Abstract

This invention relates to the field of robotics, and more particularly to a human-computer interaction system and method based on a service intelligent robot. The system includes a visual perception module for acquiring shelf image sequences to identify candidate objects; a visual screening module for performing visual confidence analysis to mark visually credible and untrustworthy targets; an active exploration module for performing vibration-based detection on untrustworthy targets to acquire physical response signals; an exploration analysis module for analyzing vibration features and extracting physical feature parameters to determine attribute feature information; a target determination module for determining the target determination result; and a grasping control module for determining a grasping confirmation command based on the determination result and adjusting baseline feature parameters based on execution feedback. This invention reduces the risk of misgrabbing in visually limited scenarios, improves the robot's accuracy in object discrimination, and enhances interaction reliability.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to a human-computer interaction system and method based on a service intelligent robot. Background Technology

[0002] With the rapid development of service robot technology, vision-based autonomous interactive systems have been widely applied to object recognition and grasping tasks in scenarios such as unmanned retail, warehouse picking, and hotel delivery. In these applications, robots need to accurately locate and retrieve the requested item from the shelf area based on the user's service request. However, due to factors such as shelf obstruction, insufficient lighting at night, or highly similar product packaging, relying solely on vision is insufficient to reliably distinguish candidate objects, easily leading to visually unreliable targets and grasping errors.

[0003] Chinese Patent Publication No. CN110722538A discloses an intelligent service robot. Its structure includes a guide rail assembly for forming a combination of straight lines and curves to facilitate the movement of other internal structures. Several sliding item distribution devices are arranged on the upper side of the guide rail assembly for distributing items. This device adopts the design of the guide rail assembly and the sliding item distribution devices. With the cooperation of various components and structures inside the device, it realizes the function of efficiently and intelligently distributing items to personnel in a certain working area, which greatly improves the practicality and reliability of the device.

[0004] However, the following problems still exist in the existing technology: 1. In existing technologies, most robots rely on vision for object recognition. When there are occluded scenes or poor lighting conditions, the accuracy of visual recognition will drop significantly, making it difficult for the robot to reliably distinguish between the desired object and the interfering object, and there is a risk of grasping errors.

[0005] 2. In the existing technology, the means by which robots acquire information about objects are relatively simple, mainly relying on visual signals for judgment. Robots lack other dimensions of perception to help confirm the true attributes of objects, making it difficult to accurately distinguish objects that look similar but have different materials, which affects the reliability of the interaction process. Summary of the Invention

[0006] To address this issue, the present invention provides a human-computer interaction system and method based on a service intelligent robot, which overcomes the problem that existing technologies have limited robot perception methods and insufficient visual reliability under occlusion or low light conditions, making it difficult for robots to accurately confirm the true attributes of objects and resulting in poor interaction reliability.

[0007] To achieve the above objectives, the present invention provides a human-computer interaction system based on a service intelligent robot, comprising: The visual perception module is used to respond to service requests for obtaining the target, acquire image sequences of the target shelf area at a predetermined sampling period, and determine each candidate object and its corresponding sub-image information. The visual screening module is used to perform visual confidence analysis on each candidate object based on the sub-image information corresponding to each candidate object, and to mark the candidate object as a visually believable target or a visually unbelievable target. An active detection module is used to perform vibration detection on a target shelf for visually unreliable targets and to acquire the physical response signal generated by the target shelf in response to the vibration detection. The detection and analysis module is used to perform vibration feature analysis based on the physical response signal to extract physical feature parameters, and compare and match the physical feature parameters with the reference feature parameters to determine the attribute feature information of the visually untrustworthy target. The target determination module is used to determine the type of each candidate object based on the visually credible target, the visually untrustworthy target, and the attribute feature information, so as to determine the target determination result. The grasping control module is used to determine the grasping confirmation instruction based on the target determination result, obtain instruction execution feedback, and adjust the baseline feature parameters.

[0008] Furthermore, the visual perception module includes: An image acquisition unit is used to acquire an image sequence of the target shelf area at the predetermined sampling period in response to a service request; The object extraction unit is used to process each frame of the image sequence, segment each candidate object from the target shelf area, and determine the sub-image information corresponding to each candidate object.

[0009] Furthermore, the visual screening module performs visual confidence analysis on each candidate object based on the sub-image information corresponding to each candidate object, and the process of marking the candidate object as a visually believable target or a visually unbelievable target includes, Based on the sub-image information corresponding to each candidate object, the visual confidence value of each candidate object is determined; If the visual confidence value is greater than or equal to the preset visual confidence threshold, the corresponding candidate object is marked as a visually credible target. If the visual confidence value is less than the preset visual confidence threshold, the corresponding candidate object is marked as a visually untrustworthy target.

[0010] Furthermore, the active exploration module includes: a vibration control unit, which controls the service robot to move to the target shelf area for visually untrusted targets, and controls the vibration excitation device on the service robot to apply standardized mechanical excitation to the shelf surface; An echo acquisition unit is used to acquire the echo signal of structural vibration generated on the shelf surface in response to the standardized mechanical excitation. The physical response signal is the structural vibration echo signal generated by the object in response to the standardized mechanical excitation.

[0011] Furthermore, the exploration and analysis module includes, A feature extraction unit is used to perform vibration feature analysis based on the physical response signal in order to extract physical feature parameters; The attribute matching unit is used to compare and match the physical feature parameters with the baseline feature parameters of each category of targets in the preset feature database, and determine the attribute feature information of the visually untrustworthy target based on the matching result.

[0012] Furthermore, the process by which the feature extraction unit performs vibration feature analysis based on the physical response signal to extract physical feature parameters includes: determining the vibration decay time and the vibration peak frequency based on the physical response signal; The mass parameters are determined based on the vibration decay time. Material parameters are determined based on the peak vibration frequency; The physical characteristic parameters include mass parameters and material parameters.

[0013] Furthermore, the target determination module includes: a credible target determination unit, which is used to determine the type of the visually credible target based on the sub-image information corresponding to the visually credible target, and determine the target determination result of the visually credible target; The untrusted target determination unit is used to determine the type of the visually untrusted target based on the attribute feature information of the visually untrusted target, and to determine the target determination result of the untrusted target.

[0014] Furthermore, the grasping control module includes: an instruction determination unit, which is used to determine the target grasping object from each candidate object based on the target determination result, and to determine the grasping confirmation instruction based on the sub-image information corresponding to the target grasping object; The feedback adjustment unit is used to obtain the execution feedback of the capture confirmation instruction and adjust the baseline feature parameters based on the execution feedback.

[0015] Furthermore, the process of the feedback adjustment unit obtaining the execution feedback of the capture confirmation instruction and adjusting the benchmark feature parameters based on the execution feedback includes: if the execution feedback is correct execution feedback, then the benchmark feature parameters are not corrected; If the execution feedback is an error execution feedback, then based on the corresponding physical characteristic parameters, the benchmark characteristic parameters of the corresponding material category in the benchmark characteristic parameters are corrected; In this context, the correctly executed feedback indicates that the grasped object is identified as a desired target, while the incorrectly executed feedback indicates that the grasped object is identified as a non-desired target.

[0016] On the other hand, a method for a human-computer interaction system based on service intelligent robots is also provided, characterized by including, In response to a service request to obtain the target, an image sequence of the target shelf area is acquired at a predetermined sampling period to determine each candidate object and its corresponding sub-image information; Based on the sub-image information corresponding to each candidate object, visual confidence analysis is performed on each candidate object, and the candidate object is marked as a visually credible target or a visually untrustworthy target. For visually unreliable targets, vibration detection is performed on the target shelf to obtain the physical response signal generated by the target shelf in response to the vibration detection; Based on the physical response signal, vibration feature analysis is performed, physical feature parameters are extracted, and the physical feature parameters are compared and matched with the benchmark feature parameters to determine the attribute feature information of the visually untrustworthy target. Based on the visually credible target, the visually untrustworthy target, and the attribute feature information, the category of each candidate object is determined, and the target determination result is determined. Based on the target determination result, a capture confirmation instruction is determined, instruction execution feedback is obtained, and the baseline feature parameters are adjusted based on the execution feedback.

[0017] Compared with existing technologies, the system of this invention includes a visual perception module for acquiring shelf image sequences to identify candidate objects; a visual screening module for performing visual confidence analysis to mark visually credible and untrustworthy targets; an active exploration module for performing vibration-based detection on untrustworthy targets to acquire physical response signals; an exploration analysis module for performing vibration feature analysis to extract physical feature parameters to determine attribute feature information; a target determination module for determining the target determination result; and a grasping control module for determining the grasping confirmation command based on the determination result and adjusting the baseline feature parameters based on execution feedback. This invention reduces the risk of misgrabbing in visually limited scenarios, improves the robot's accuracy in object discrimination, and enhances interaction reliability.

[0018] In particular, this invention considers the practical situation where visual reliability decreases under occlusion or low-light conditions. In scenarios such as unmanned retail, shelf obstruction or insufficient nighttime lighting often leads to insufficient visual confidence in some candidate objects. This invention uses an active detection module to perform vibration-based detection on visually untrusted targets, collecting physical response signals. The detection and analysis module extracts mass and material parameters as physical feature parameters, which are then compared and matched with benchmark feature parameters to determine attribute feature information. This provides visually untrusted targets with a second dimension of perceptual data support beyond visual perception. The vibration-based detection mechanism effectively fills in blind spots in visually limited scenarios, preventing abandonment of grasping due to visual failure and reducing the probability of false grasping.

[0019] In particular, this invention uses a target determination module to comprehensively determine the credibility of visually credible and visually untrustworthy targets. In the working scenarios of service robots, both visually credible and untrustworthy candidate objects often exist simultaneously within the field of view. In scenarios where the requested item is obscured by objects in front, relying solely on visually credible targets for determination may lead to task failure and a poor interactive experience. This invention determines the category of visually credible targets based on sub-image information and visually untrustworthy targets based on attribute feature information, ultimately combining the two determination results to determine the target determination result. This avoids the problem of requested items being missed due to occlusion, improving the reliability of the service robot's interaction process.

[0020] In particular, this invention introduces an adaptive adjustment mechanism based on instruction execution feedback through the grasping control module. In practical applications, the baseline feature parameters in the material feature database cannot perfectly cover the true physical properties of all products. Static matching rules are prone to attribute misjudgment when product batches change, leading to a continuous decline in matching accuracy. When the execution feedback is an erroneous execution feedback, this invention corrects the baseline feature parameters of the corresponding material category based on the current physical feature parameters and matching failure information. This mechanism enables the material feature database to be continuously expanded and calibrated during actual operation, improving the long-term matching reliability and environmental adaptability of the system. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of a human-computer interaction system based on a service intelligent robot according to an embodiment of the present invention; Figure 2 A logic diagram for marking visually believable targets and visually unbelievable targets in an embodiment of the present invention; Figure 3 This is a logic diagram illustrating how the baseline feature parameters are adjusted based on the execution feedback in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the steps of a human-computer interaction method based on a service intelligent robot, as described in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0023] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0024] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0025] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0026] Please see Figure 1 As shown, it is a structural schematic diagram of a human-computer interaction system based on a service intelligent robot according to an embodiment of the present invention. The human-computer interaction system based on a service intelligent robot of the present invention includes: The visual perception module is used to respond to service requests for obtaining the target, acquire image sequences of the target shelf area at a predetermined sampling period, and determine each candidate object and its corresponding sub-image information. The visual screening module is used to perform visual confidence analysis on each candidate object based on the sub-image information corresponding to each candidate object, and to mark the candidate object as a visually believable target or a visually unbelievable target. An active detection module is used to perform vibration detection on a target shelf for visually unreliable targets and to acquire the physical response signal generated by the target shelf in response to the vibration detection. The detection and analysis module is used to perform vibration feature analysis based on the physical response signal to extract physical feature parameters, and compare and match the physical feature parameters with the reference feature parameters to determine the attribute feature information of the visually untrustworthy target. The target determination module is used to determine the type of each candidate object based on the visually credible target, the visually untrustworthy target, and the attribute feature information, so as to determine the target determination result. The grasping control module is used to determine the grasping confirmation instruction based on the target determination result, obtain instruction execution feedback, and adjust the baseline feature parameters.

[0027] In this embodiment of the invention, the service intelligent robot is applicable to unmanned retail scenarios such as nighttime unmanned pharmacies and chain supermarkets. After a customer initiates a request to retrieve an item, the robot moves to the target shelf area, where a visual perception module collects images of the shelf and identifies candidate objects. For candidate objects obscured by the shelf or located in low-light areas, visual recognition results are often unreliable, and these objects are marked as visually untrustworthy targets. In this case, the robot does not abandon the request directly but controls its end-effector to apply contact excitation to the shelf surface, collecting vibration echo signals returned from the shelf surface, thereby obtaining a second dimension of information beyond vision, which serves as a basis for identifying the product category.

[0028] Specifically, this invention adds a contact-based vibration sensing path outside the visual perception link. After the visual screening module marks candidate objects as visually believable and visually unbelievable targets based on visual confidence analysis, the target determination module determines the type of visually believable targets based on sub-image information. For visually unbelievable targets, the active detection module controls the end-point detection device to perform standardized mechanical excitation on the shelf surface, collecting structural vibration echo signals as physical response signals. The detection and analysis module extracts the vibration decay time and peak frequency from this signal, determines the mass parameters and material parameters respectively, as physical feature parameters, and compares and matches them with the benchmark feature parameters of each type of material in the preset material feature database to obtain the surface material information and mass information of the object. The target determination module summarizes the type determination results of visually believable targets and the attribute feature determination results of visually unbelievable targets to form a comprehensive confidence determination of each candidate object, thus determining the target determination result.

[0029] Specifically, the grasping control module determines the grasping confirmation command based on the target determination result. After receiving the command execution feedback, if the feedback is correct, the current baseline feature parameters are retained; if the feedback is incorrect, the baseline feature parameters of the corresponding material category are corrected based on the physical feature parameters used in the current matching and the matching failure information. This invention overcomes the reliability deficiency of single visual perception in occluded or low-light scenarios by coordinating two pathways: visual perception and contact vibration perception. Simultaneously, by continuously calibrating the baseline feature parameters through execution feedback, the system can gradually adapt to the real physical properties of the goods on site, improving the accuracy of the service robot's object identification and the reliability of its interaction.

[0030] It is understood that the embodiments of the present invention are applicable to merchandise sales scenarios such as unmanned retail at night where lighting conditions are uncontrollable and shelf layouts are compact. Furthermore, the technical approach of the present invention can also be extended to various situations where it is necessary to distinguish the material or quality of objects but visual perception conditions are limited, such as the identification and classification of packaging boxes of different materials in warehouse picking scenarios, and the differentiation of different packaged goods in fresh food delivery scenarios. As service intelligent robots enter complex operational fields, the contact-based vibration sensing and visual collaborative working scheme of the embodiments of the present invention can also provide technical support for the interaction reliability of service robots under visually limited conditions in other scenarios, which will not be elaborated further here.

[0031] Specifically, there are no restrictions on the specific forms of the object extraction unit, visual screening module, vibration control unit, detection and analysis module, target determination module, and grasping control module. They can be composed of logic components, including field-programmable processors, computers, or microprocessors within computers, which will not be elaborated further. Specifically, the visual perception module includes: An image acquisition unit is used to acquire an image sequence of the target shelf area at the predetermined sampling period in response to a service request; The object extraction unit is used to process each frame of the image sequence, segment each candidate object from the target shelf area, and determine the sub-image information corresponding to each candidate object.

[0032] Specifically, the image acquisition unit is an industrial camera module deployed on the service intelligent robot, which is used to continuously take pictures of the target shelf area according to a predetermined sampling period after receiving a service request, and to obtain an image sequence containing each candidate object.

[0033] In this embodiment of the invention, an 8.1-megapixel depth camera is selected as the image acquisition unit. In the implementation of this embodiment, the performance of the image acquisition unit should meet the requirement of clearly acquiring images of the shelf area under low-light conditions and providing a sufficiently high-resolution image sequence for the subsequent object extraction unit; this will not be elaborated further.

[0034] Specifically, the predetermined sampling period is the time interval during which the service robot collects images of the target shelf area after receiving a service request. The image sequence consists of multiple frames of shelf images continuously collected according to the sampling period in a single service task. The purpose of collecting multiple frames of images at the sampling period to form a sequence is to provide the visual screening module with multiple sets of sub-image information of the same shelf area at different collection times. This facilitates the improvement of the reliability of candidate object recognition and segmentation through multi-frame comparison and reduces missed detections and false detections caused by momentary occlusion due to local reflections in a single frame image.

[0035] In this embodiment of the invention, the predetermined sampling period ranges from [0.5 seconds to 1.5 seconds], and the number of frames in the image sequence ranges from [30 frames to 120 frames]. If the sampling period is too short or the number of frames is too small, the information obtained from multi-frame comparison is limited; if the sampling period is too long, the object recognition takes too long, reducing the interactive experience.

[0036] Specifically, the sub-image information refers to the local image regions corresponding to each candidate object and their spatial location information within the shelf, segmented from the target shelf area by the object extraction unit after processing each frame of the image sequence. The significance of the sub-image information lies in providing the visual screening module with independent visual data slices for each candidate object, enabling visual confidence analysis to be performed individually for each candidate object, thus avoiding interference between image features of different objects.

[0037] It is understood that the process of the object extraction unit acquiring sub-image information can be implemented using existing publicly available technologies, such as instance segmentation algorithms based on deep learning or object detection algorithms based on region convolutional neural networks. In the implementation of this example, any existing image segmentation technology that can accurately segment each candidate object from the image sequence and determine the corresponding sub-image information can be selected, and will not be elaborated further here.

[0038] Please see Figure 2 As shown, this is a logical judgment diagram for marking visually believable targets and visually unbelievable targets according to an embodiment of the present invention. The visual screening module performs visual confidence analysis on each candidate object based on the sub-image information corresponding to each candidate object. The process of marking the candidate object as a visually believable target or a visually unbelievable target includes... Based on the sub-image information corresponding to each candidate object, the visual confidence value of each candidate object is determined; If the visual confidence value is greater than or equal to the preset visual confidence threshold, the corresponding candidate object is marked as a visually credible target. If the visual confidence value is less than the preset visual confidence threshold, the corresponding candidate object is marked as a visually untrustworthy target.

[0039] Specifically, the process of determining the visual confidence value of each candidate object based on the sub-image information corresponding to each candidate object includes: Specifically, the process of determining the visual confidence value of each candidate object based on the sub-image information corresponding to each candidate object includes: The sub-image information of each candidate object is compared with the product images in the preset product image database to determine the matching similarity between each candidate object and each category of product in the database. The highest matching similarity for each candidate object is used as the visual confidence value of that candidate object.

[0040] Specifically, the matching similarity is a quantitative value of similarity obtained by comparing the sub-image information of the candidate object with the product images of each category in the product image database. It is used to measure the degree of visual matching between the candidate object and a certain category of products in the database. In this embodiment of the invention, the output value of the matching similarity ranges from [0, 1]. The closer the value is to 1, the higher the degree of visual matching between the candidate object and the corresponding category of products.

[0041] In this embodiment of the invention, the calculation of matching similarity can be implemented using existing publicly available techniques, such as mapping to the [0, 1] interval based on the Euclidean distance metric of feature descriptors, or outputting similarity values ​​within the [0, 1] interval based on the cosine similarity of depth feature vectors. In the implementation of this embodiment, the selected technique is sufficient to accurately compare the features of candidate object sub-images with product images in the database and output matching similarity values, which will not be elaborated further.

[0042] Specifically, the visual confidence threshold is preset by the actual application scenario of the embodiment based on the recognition accuracy of the visual perception module and the load capacity of the active exploration module. In this embodiment of the invention, the visual confidence threshold ranges from [0.85, 0.95].

[0043] If the visual confidence threshold is too small, the visual screening module will mark too many candidate objects with unclear visual features as visually credible targets, which will increase the false detection rate of the target determination module and increase the risk of erroneous grasping by the subsequent grasping control module. If the visual confidence threshold is too high, the visual screening module will mark a large number of candidate objects that could be accurately identified by vision as visually unreliable targets, which will increase the number of vibration detections by the active detection module, reduce interaction efficiency, and may aggravate the mechanical wear of the end detection device.

[0044] It is understandable that setting the visual confidence threshold within the range of [0.85, 0.95] allows the visual screening module to ensure high recognition accuracy of visually credible targets while reasonably diverting candidate objects with blurred visual features due to shelf obstruction, low lighting conditions, or highly similar product packaging to the active detection module for vibration feature verification. This achieves optimal load distribution between the visual and vibration sensing pathways, reducing the overall probability of false grabbing in the system.

[0045] Specifically, the active exploration module includes: a vibration control unit, which controls the service robot to move to the target shelf area for visually unreliable targets, and controls the vibration excitation device on the service robot to apply standardized mechanical excitation to the shelf surface; An echo acquisition unit is used to acquire the echo signal of structural vibration generated on the shelf surface in response to the standardized mechanical excitation. The physical response signal is the structural vibration echo signal generated by the object in response to the standardized mechanical excitation.

[0046] Specifically, the vibration excitation device is a contact-type vibration assembly deployed on the end effector of the service robot, used to apply controllable mechanical excitation to the shelf surface where visually unreliable targets are located after the robot arrives at the target shelf area. The device has a built-in electromagnetic drive unit that drives a striker or impact hammer to produce a single strike on the shelf surface via a short-time pulse current.

[0047] In this embodiment of the invention, a miniature electromagnetic vibrator is selected as the vibration excitation device, with a peak excitation force range of [5N, 20N] and an excitation duration range of [1ms, 5ms]. If the excitation force is too small, the echo signal will be too weak, making it difficult for subsequent analysis modules to extract effective vibration characteristic parameters; if the excitation force is too large, it may damage the shelf surface or merchandise. If the excitation duration is too short, the excitation pulse energy is insufficient to fully excite the vibration response frequency band of the shelf and merchandise; if the excitation duration is too long, the excitation signal and echo signal are prone to complex superposition phenomena, which is not conducive to the separation and extraction of the echo signal. In the implementation of this embodiment, the excitation force and excitation duration can be adjusted within the above range according to the actual shelf material and structural stiffness to ensure that a clear and usable vibration echo signal is acquired, which will not be elaborated further.

[0048] Specifically, the standardized mechanical excitation refers to a pre-set and repeatable mechanical excitation action applied to the shelf surface by the vibration excitation device. By applying a mechanical excitation action with constant excitation force, duration, and direction, the structural vibration echo signals collected from different detections are ensured to be comparable, avoiding deviations in vibration characteristic analysis results due to inconsistent excitation parameters.

[0049] In this embodiment of the invention, the excitation force of the standardized mechanical excitation is set to 10N, the excitation duration is set to 2ms, and the excitation direction is perpendicular to the shelf surface. In the implementation of this embodiment, the parameters can be adjusted within the above range according to the actual shelf material and structural stiffness to meet the quality requirements of the vibration echo signal under different shelf environments; this will not be elaborated further.

[0050] Specifically, the echo acquisition unit is a vibration signal acquisition component deployed on the end effector of the service robot. It is used to acquire the structural vibration echo signal generated by the shelf surface in response to the standardized mechanical excitation applied by the vibration excitation device. The unit has a built-in accelerometer sensor that is attached to the shelf surface or coupled to the shelf surface through a probe, converting the mechanical vibration of the shelf surface into an electrical signal output.

[0051] In this embodiment of the invention, the sampling frequency and range of the echo acquisition unit should be able to capture the waveform of the vibration echo signal on the shelf surface, and should meet the requirement of providing sufficiently clear signal data for the detection and analysis module, which will not be elaborated further.

[0052] Specifically, the exploration and analysis module includes, A feature extraction unit is used to perform vibration feature analysis based on the physical response signal in order to extract physical feature parameters; The attribute matching unit is used to compare and match the physical feature parameters with the baseline feature parameters of each category of targets in the preset feature database, and determine the attribute feature information of the visually untrustworthy target based on the matching result.

[0053] Specifically, the process by which the feature extraction unit performs vibration feature analysis based on the physical response signal to extract physical feature parameters includes: determining the vibration decay time and the vibration peak frequency based on the physical response signal; The mass parameters are determined based on the vibration decay time. Material parameters are determined based on the peak vibration frequency; The physical characteristic parameters include mass parameters and material parameters.

[0054] Specifically, the attribute matching unit compares the physical feature parameters with the baseline feature parameters of each category of targets in the preset feature database, and determines the attribute feature information of the visually untrustworthy target based on the matching result. The quality parameters of the visually untrustworthy targets are compared one by one with the benchmark quality parameters of each category of targets in the feature database to determine the quality deviation rate. The material parameters of the visually untrustworthy target are compared one by one with the baseline material parameters of each category of target in the feature database to determine the material deviation rate. If the quality deviation rate is less than the preset quality deviation threshold and the material deviation rate is less than the preset material deviation threshold, then the quality comparison matching is determined to be successful and the material comparison matching is successful, and the corresponding category is determined as a candidate matching category. The category with the smallest weighted sum of quality deviation rate and material deviation rate from the candidate matching categories is selected as the attribute feature information of the visually untrustworthy target.

[0055] It is understood that the baseline feature parameters of each category of target in the feature database need to be pre-entered before the goods are put into storage and shelving. For each category of goods displayed in the target shelf area, standard samples are selected, and the active detection module is controlled to perform vibration detection on the standard samples. The detection and analysis module extracts the corresponding physical feature parameters as the baseline feature parameters of each category of target, and the baseline feature parameters are entered into the feature database to establish a standard physical feature file for each category of goods.

[0056] Specifically, the quality deviation threshold is determined as follows: The quality deviation threshold is determined based on the allowable deviation range of the target benchmark quality parameters for each category in the feature database, and the lower limit of the allowable deviation range is taken as the quality deviation threshold. In this embodiment of the invention, the quality deviation threshold ranges from [0.03, 0.05].

[0057] Understandably, if the quality deviation threshold is too low, the attribute matching unit will be too strict in comparing the quality parameters of visually untrustworthy targets. Normal quality fluctuations caused by differences in production batches of similar products may easily lead to matching failures, resulting in visually untrustworthy targets being misjudged as unknown categories and missed, increasing the risk of task failure. If the quality deviation threshold is too high, the quality comparison will be too lenient, and the quality parameter ranges of different categories of products may easily overlap, causing the attribute matching unit to misjudge non-demand categories as candidate matching categories, increasing the probability of erroneous capture by the subsequent capture control module.

[0058] Specifically, the material deviation threshold is determined as follows: The allowable deviation range is determined based on the frequency interval mapped by the target reference material parameters of each category in the feature database, and the upper limit of the allowable deviation range is taken as the material deviation threshold. In this embodiment of the invention, the material deviation threshold ranges from [0.01, 0.03].

[0059] Understandably, if the material deviation threshold is too low, the attribute matching unit will be too strict in comparing the material parameters of visually untrustworthy targets. Small frequency shifts caused by vibration transmission noise on the shelf surface or measurement errors of the vibration pickup unit may easily cause the comparison and matching to fail, resulting in visually untrustworthy targets being incorrectly classified into unknown categories. If the material deviation threshold is too high, the material comparison will be too lenient. There will be some overlap in the vibration frequency ranges of different material categories, which may cause the attribute matching unit to misjudge non-demand categories with large material differences as candidate matching categories.

[0060] Specifically, the target determination module includes: a credible target determination unit, which is used to determine the type of the visually credible target based on the sub-image information corresponding to the visually credible target, and determine the target determination result of the visually credible target; The untrusted target determination unit is used to determine the type of the visually untrusted target based on the attribute feature information of the visually untrusted target, and to determine the target determination result of the untrusted target.

[0061] Specifically, the target determination result of visually credible targets can be achieved using existing commodity image recognition technology, that is, the commodity category to which the target belongs is determined based on the matching result between the sub-image information and the commodity image database, which will not be elaborated further.

[0062] Specifically, the determination of an untrusted target involves comparing the physical feature parameters of the untrusted target with the baseline feature parameters of each category of targets in the feature database one by one, and selecting the candidate category whose baseline feature parameters are closest to the physical feature parameters of the current untrusted target as the determination result of the untrusted target.

[0063] Specifically, the grasping control module includes: an instruction determination unit, which is used to determine the target grasping object from each candidate object based on the target determination result, and to determine the grasping confirmation instruction based on the sub-image information corresponding to the target grasping object; The feedback adjustment unit is used to obtain the execution feedback of the capture confirmation instruction and adjust the baseline feature parameters based on the execution feedback.

[0064] Please see Figure 3 As shown, it is a logic judgment diagram of adjusting the benchmark feature parameters based on the execution feedback in an embodiment of the present invention. The feedback adjustment unit obtains the execution feedback of the capture confirmation instruction. The process of adjusting the benchmark feature parameters based on the execution feedback includes: if the execution feedback is correct execution feedback, then the benchmark feature parameters are not corrected. If the execution feedback is an error execution feedback, then based on the corresponding physical characteristic parameters, the benchmark characteristic parameters of the corresponding material category in the benchmark characteristic parameters are corrected; In this context, the correctly executed feedback indicates that the grasped object is identified as a desired target, while the incorrectly executed feedback indicates that the grasped object is identified as a non-desired target.

[0065] Specifically, the execution feedback refers to the confirmation information obtained by the grasping control module after issuing a grasping confirmation command and completing the grasping action, regarding whether the grasped object is confirmed as the desired target. In unmanned retail scenarios, confirmation information can be obtained through customer selection confirmation on the interactive terminal, detection of customer's actions in taking delivered goods, or customer-initiated return / exchange requests. Execution feedback is divided into two categories: correct execution feedback and incorrect execution feedback. Correct execution feedback indicates that the grasped object has been confirmed as the desired target, while incorrect execution feedback indicates that the grasped object has been confirmed as a non-desired target.

[0066] In this embodiment of the invention, the process of correcting the baseline feature parameters corresponding to the material category in the baseline feature parameters based on the physical feature parameters used in the current matching includes: If the execution feedback is an error, the physical characteristic parameters measured in this instance will be added to the standard physical characteristic file of the corresponding material category in the characteristic database.

[0067] Specifically, the aforementioned feedback adjustment process plays a crucial role in the overall solution by addressing the discrepancy between the initial calibration values ​​of the feature database and the actual physical attributes of the goods on-site. In the initial stages of system deployment, the baseline feature parameters for each category of goods in the feature database are determined only by a small number of calibration samples. When the packaging material or quality of goods on the shelf fluctuates due to batch changes, the initial baseline values ​​may deviate from the actual items, thus affecting the accuracy of the unreliable target identification unit. By supplementing the measured physical feature parameters into the correct category's sample set each time an error feedback occurs, the baseline feature parameters can gradually approach the actual physical attribute values ​​of the goods in the current shelf environment.

[0068] Understandably, as error feedback is continuously triggered and samples are expanded during system operation, the baseline feature parameters of each product category gradually accumulate from a small initial number of calibration samples to statistical reference values ​​containing multiple measured data. Matching accuracy gradually improves with the number of services. This process avoids the continuous degradation of matching performance due to batch changes in product packaging or changes in shelf environment, reduces the frequency of manual recalibration during long-term system operation, and controls the deployment and maintenance costs of service robots in unmanned retail scenarios.

[0069] On the other hand, embodiments of the present invention also provide a method for a human-computer interaction system based on a service intelligent robot. Please refer to [link to relevant documentation]. Figure 4 The diagram shown is a flowchart illustrating the steps of a human-computer interaction method based on a service intelligent robot according to an embodiment of the present invention. The method implemented by the present invention includes: Step S1: In response to the service request to obtain the target, acquire the image sequence of the target shelf area at a predetermined sampling period, and determine each candidate object and its corresponding sub-image information; Step S2: Based on the sub-image information corresponding to each candidate object, perform visual confidence analysis on each candidate object and mark the candidate object as a visually credible target or a visually untrustworthy target. Step S3: For visually unreliable targets, perform vibration detection on the target shelf and obtain the physical response signal generated by the target shelf in response to the vibration detection; Step S4: Based on the physical response signal, perform vibration feature analysis, extract physical feature parameters, compare and match the physical feature parameters with the benchmark feature parameters, and determine the attribute feature information of the visually untrustworthy target. Step S5: Based on the visually credible target, the visually untrustworthy target, and the attribute feature information, determine the category of each candidate object and determine the target determination result; Step S6: Based on the target determination result, determine the capture confirmation instruction, obtain instruction execution feedback, and adjust the baseline feature parameters based on the execution feedback.

[0070] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A human-computer interaction system based on a service intelligent robot, characterized in that, include: The visual perception module is used to respond to service requests for obtaining the target, acquire image sequences of the target shelf area at a predetermined sampling period, and determine each candidate object and its corresponding sub-image information. The visual screening module is used to perform visual confidence analysis on each candidate object based on the sub-image information corresponding to each candidate object, and to mark the candidate object as a visually believable target or a visually unbelievable target. An active detection module is used to perform vibration detection on a target shelf for visually unreliable targets and to acquire the physical response signal generated by the target shelf in response to the vibration detection. The detection and analysis module is used to perform vibration feature analysis based on the physical response signal to extract physical feature parameters, and compare and match the physical feature parameters with the reference feature parameters to determine the attribute feature information of the visually untrustworthy target. The target determination module is used to determine the type of each candidate object based on the visually credible target, the visually untrustworthy target, and the attribute feature information, so as to determine the target determination result. The grasping control module is used to determine the grasping confirmation instruction based on the target determination result, obtain instruction execution feedback, and adjust the baseline feature parameters.

2. The human-computer interaction system based on a service intelligent robot according to claim 1, characterized in that, The visual perception module includes: An image acquisition unit is used to acquire an image sequence of the target shelf area at the predetermined sampling period in response to a service request; The object extraction unit is used to process each frame of the image sequence, segment each candidate object from the target shelf area, and determine the sub-image information corresponding to each candidate object.

3. The human-computer interaction system based on a service intelligent robot according to claim 1, characterized in that, The visual screening module performs visual confidence analysis on each candidate object based on the sub-image information corresponding to each candidate object, and the process of marking the candidate object as a visually believable target or a visually unbelievable target includes: Based on the sub-image information corresponding to each candidate object, the visual confidence value of each candidate object is determined; If the visual confidence value is greater than or equal to the preset visual confidence threshold, the corresponding candidate object is marked as a visually credible target. If the visual confidence value is less than the preset visual confidence threshold, the corresponding candidate object is marked as a visually untrustworthy target.

4. The human-computer interaction system based on a service intelligent robot according to claim 1, characterized in that, The active detection module includes: a vibration control unit, which controls the service intelligent robot to move to the target shelf area for visually unreliable targets, and controls the vibration excitation device on the service intelligent robot to apply standardized mechanical excitation to the shelf surface; An echo acquisition unit is used to acquire the echo signal of structural vibration generated on the shelf surface in response to the standardized mechanical excitation. The physical response signal is the structural vibration echo signal generated by the object in response to the standardized mechanical excitation.

5. The human-computer interaction system based on a service intelligent robot according to claim 1, characterized in that, The detection and analysis module includes, A feature extraction unit is used to perform vibration feature analysis based on the physical response signal in order to extract physical feature parameters; The attribute matching unit is used to compare and match the physical feature parameters with the baseline feature parameters of each category of targets in the preset feature database, and determine the attribute feature information of the visually untrustworthy target based on the matching result.

6. The human-computer interaction system based on a service intelligent robot according to claim 5, characterized in that, The process by which the feature extraction unit performs vibration feature analysis based on the physical response signal to extract physical feature parameters includes: determining the vibration decay time and the vibration peak frequency based on the physical response signal; The mass parameters are determined based on the vibration decay time. Material parameters are determined based on the peak vibration frequency; The physical characteristic parameters include mass parameters and material parameters.

7. The human-computer interaction system based on a service intelligent robot according to claim 1, characterized in that, The target determination module includes: a credible target determination unit, which is used to determine the type of the visually credible target based on the sub-image information corresponding to the visually credible target, and determine the target determination result of the visually credible target; The untrusted target determination unit is used to determine the type of the visually untrusted target based on the attribute feature information of the visually untrusted target, and to determine the target determination result of the untrusted target.

8. The human-computer interaction system based on a service intelligent robot according to claim 1, characterized in that, The grasping control module includes: an instruction determination unit, which is used to determine the target grasping object from each candidate object based on the target determination result, and to determine the grasping confirmation instruction based on the sub-image information corresponding to the target grasping object; The feedback adjustment unit is used to obtain the execution feedback of the capture confirmation instruction and adjust the baseline feature parameters based on the execution feedback.

9. The human-computer interaction system based on a service intelligent robot according to claim 8, characterized in that, The process of the feedback adjustment unit obtaining the execution feedback of the capture confirmation instruction and adjusting the benchmark feature parameters based on the execution feedback includes: if the execution feedback is correct execution feedback, then the benchmark feature parameters are not corrected; If the execution feedback is an error execution feedback, then based on the corresponding physical characteristic parameters, the benchmark characteristic parameters of the corresponding material category in the benchmark characteristic parameters are corrected; In this context, the correctly executed feedback indicates that the grasped object is identified as a desired target, while the incorrectly executed feedback indicates that the grasped object is identified as a non-desired target.

10. A method applied to a human-computer interaction system based on a service intelligent robot as described in any one of claims 1-9, characterized in that, include, In response to a service request to obtain the target, an image sequence of the target shelf area is acquired at a predetermined sampling period to determine each candidate object and its corresponding sub-image information; Based on the sub-image information corresponding to each candidate object, visual confidence analysis is performed on each candidate object, and the candidate object is marked as a visually credible target or a visually untrustworthy target. For visually unreliable targets, vibration detection is performed on the target shelf to obtain the physical response signal generated by the target shelf in response to the vibration detection; Based on the physical response signal, vibration feature analysis is performed, physical feature parameters are extracted, and the physical feature parameters are compared and matched with the benchmark feature parameters to determine the attribute feature information of the visually untrustworthy target. Based on the visually credible target, the visually untrustworthy target, and the attribute feature information, the category of each candidate object is determined, and the target determination result is determined. Based on the target determination result, a capture confirmation instruction is determined, instruction execution feedback is obtained, and the baseline feature parameters are adjusted based on the execution feedback.

Images