A multi-modal perception contact risk response method, apparatus, device, storage medium and program product

Abstract

This disclosure provides a multimodal sensing contact risk response method, apparatus, device, storage medium, and program product. The method includes determining the corresponding distal high-value organ site and its connecting link based on the target mature fruit site; determining candidate approach paths based on the target mature fruit site and the pose of the harvesting robotic arm; obtaining the pre-contact reference state of the distal high-value organ and the connecting link based on the contact risk segment in the candidate approach path; controlling the harvesting robotic arm to execute preset commands based on the collected contact signals; and obtaining the direct response of the distal high-value organ and the alternative response of the connecting link within a preset time window determined by the preset commands, performing a distal propagation risk assessment, obtaining a risk assessment value and an adjusted risk index value, comparing the assessment result with a preset risk acceptance level, and determining the execution result of the contact risk segment. It can determine whether the path can continue based on the distal response when slight contact or a contact trend occurs in the contact risk segment.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent harvesting technology, and in particular to a multimodal sensing contact risk response method, apparatus, device, storage medium, and program product. Background Technology

[0002] When the robotic arm approaches the target mature fruit, even slight contact with the leaves, petioles, twigs, or fruit stalks can cause disturbances to spread along the fruit stalks, forking nodes, or twigs to distant immature fruits, buds, flower buds, tender shoots, or key forking nodes, causing the distant organs to swing, be disturbed, bend, or be damaged.

[0003] In the dense canopy of plants, buds, flower buds, and tender shoots are often obscured by leaves or fruits, making it difficult for existing vision systems to reliably and directly track the movement of all minute organs. If direct observation of minute organs is considered a necessary condition for path continuation, it can easily lead to repeated retraction or path searching by the robotic arm, reducing harvesting efficiency. Complex active micro-detection actions place high demands on both hardware and software, making them unsuitable as the main control process for field harvesting. Existing technologies primarily assess local collisions or current contact forces, lacking an assessment of the risk of local contact propagating to distant high-value organs. Even when observations are unreliable, the path may still be executed, leading to a high risk of misjudging distant high-value organs. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a multimodal sensing contact risk response method, apparatus, device, storage medium, and program product.

[0005] This disclosure provides a multimodal sensing contact risk response method, the method comprising: Obtain the target mature fruit site, determine the connection link between the target mature fruit site and the distal high-value organ site within a first preset range, and determine the candidate approach path based on the target mature fruit site and the posture of the picking robot arm. The candidate approach path includes a contact risk segment. Based on the contact risk segment, the pre-contact reference state of the distal high-value organ and the connection link is obtained. In response to the contact signal obtained by the picking robot arm within the contact risk segment, the picking robot arm is controlled to execute preset instructions according to the contact signal. The preset instructions include a pause instruction and a retraction instruction. Within a preset time window determined according to the preset instruction, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, a remote propagation risk assessment is performed to obtain the assessment result, which includes a risk assessment value and an adjusted risk indicator value. The assessment results are compared with the preset risk tolerance, and the execution result of the exposure risk segment is determined based on the comparison results.

[0006] The method provided in this disclosure, comprising: acquiring a target mature fruit locus; determining a connection link between a distal high-value organ locus within a first preset range and the distal high-value organ locus based on the target mature fruit locus; and determining candidate approach paths based on the target mature fruit locus and the pose of a harvesting robotic arm, including: Obtain point cloud information of the target mature fruit, and determine the location of the target mature fruit based on the point cloud information; Based on the target mature fruit locus, expand outward to a first preset range, identify a preset high-value organ type within the first preset range, determine the feature points of the high-value organ type from the type feature table, and determine the distal high-value organ locus based on the feature points. The connection link of the distant high-value organ site is determined based on the distant high-value organ site and the target mature fruit site; Starting from the pose of the picking robotic arm and ending at the target ripe fruit location, at least one initial approach path is determined. Based on preset distance conditions, the contact risk segment in each initial approach path is determined. The path score of the corresponding initial approach path is calculated based on the contact risk segment. The initial approach path with the highest score among the path scores is selected as the candidate approach path.

[0007] The method provided in this disclosure, based on the contact risk segment, obtains a pre-contact reference state between the distal high-value organ and the connection link, responds to a contact signal obtained by the harvesting robotic arm within the contact risk segment, and controls the harvesting robotic arm to execute preset instructions according to the contact signal, including: Before the harvesting robotic arm enters the contact risk segment of the candidate approach path, the pre-contact reference state of the distal high-value organ and the connection link is collected. Based on the contact signals acquired by the harvesting robotic arm within the contact risk zone, a contact confirmation index is calculated. Contact safety boundary indicators are determined based on historical harvesting boundary tables. Compare the contact confirmation index with a preset index threshold to obtain a first comparison result, and match a pause command from a preset command table based on the first comparison result; Compare the contact safety boundary index with a preset boundary threshold to obtain a second comparison result, and match a withdrawal instruction from the preset instruction table based on the second comparison result; The harvesting robotic arm is controlled according to the pause command and the retraction command.

[0008] The method provided in this disclosure acquires the direct response of the remote high-value organ and the alternative response of the connection link within a preset time window determined according to the preset instruction. Based on the direct response and the alternative response, a remote propagation risk assessment is performed to obtain an assessment result, which includes a risk assessment value and an adjusted risk indicator value. The preset time window is activated simultaneously with the execution of the preset instruction, including: Within the preset time window, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation are calculated. Based on the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation, the credibility of observation of the remote high-value organ is calculated. If the observation confidence level is greater than or equal to the preset confidence level threshold, then a remote propagation risk assessment is performed based on the direct response to obtain a risk assessment value. If the observation confidence is less than the preset confidence threshold, and one of the force observation confidence, tactile observation confidence, and alternative observation confidence is greater than or equal to the corresponding preset threshold, then a remote propagation risk assessment is performed based on the alternative response to obtain a risk assessment value. If the observation confidence is less than a preset confidence threshold, and the confidence of force observation, tactile observation, and alternative observation are all less than their corresponding preset thresholds, then the risk indicators of the direct response and alternative response are recalculated using conservative risk adjustment. If the risk indicator is within the preset risk range, the adjusted risk indicator value is output. If the risk indicator is outside the preset risk range, then the contact risk segment is replaced.

[0009] The method provided in this disclosure, wherein the remote propagation risk assessment includes: The corresponding remote response vector is determined based on the direct response or alternative response. The corresponding response baseline quantity is matched from the standard range table based on the remote response vector. The standardized response vector of the corresponding remote high-value organ and the connection link is calculated based on the response baseline quantity and the remote response quantity. The risk assessment value is calculated based on the standardized response vector.

[0010] The method provided in this disclosure compares the assessment result with a preset risk acceptance level, and determines the execution result of the exposure risk segment based on the comparison result. The execution result includes replacing the contact risk segment and passing through the contact risk segment, including: The comparison results are obtained by comparing the assessment results with the preset risk acceptance level; If the comparison result is that the assessment result is greater than the risk tolerance, then the contact risk section is replaced; if the comparison result is that the assessment result is less than or equal to the risk tolerance, then the harvesting robotic arm is controlled to pass through the contact risk section.

[0011] This disclosure also provides a multimodal sensing contact risk response device, the device comprising: The first acquisition module is used to acquire the target mature fruit site, determine the connection link between the target mature fruit site and the distal high-value organ site within a first preset range based on the target mature fruit site, and determine the candidate approach path based on the target mature fruit site and the posture of the picking robot arm. The candidate approach path includes a contact risk segment. The second acquisition module is used to acquire the pre-contact reference state of the distal high-value organ and the connection link based on the contact risk segment, and in response to the contact signal acquired by the picking robot arm in the contact risk segment, control the picking robot arm to execute preset instructions according to the contact signal, the preset instructions including a pause instruction and a retraction instruction; The assessment module is used to acquire the direct response of the remote high-value organ and the alternative response of the connection link within a preset time window determined according to the preset instruction, and to conduct a remote propagation risk assessment based on the direct response and the alternative response to obtain an assessment result, which includes a risk assessment value and an adjusted risk indicator value. The comparison module is used to compare the assessment results with a preset risk tolerance level, and determine the execution result of the exposure risk segment based on the comparison results.

[0012] The apparatus provided in this disclosure embodiment, wherein the first acquisition module is specifically used for: Obtain point cloud information of the target mature fruit, and determine the location of the target mature fruit based on the point cloud information; Based on the target mature fruit locus, expand outward to a first preset range, identify a preset high-value organ type within the first preset range, determine the feature points of the high-value organ type from the type feature table, and determine the distal high-value organ locus based on the feature points. The connection link of the distant high-value organ site is determined based on the distant high-value organ site and the target mature fruit site; Starting from the pose of the picking robotic arm and ending at the target ripe fruit location, at least one initial approach path is determined. Based on preset distance conditions, the contact risk segment in each initial approach path is determined. The path score of the corresponding initial approach path is calculated based on the contact risk segment. The initial approach path with the highest score among the path scores is selected as the candidate approach path.

[0013] The apparatus provided in this disclosure embodiment, wherein the second acquisition module is specifically used for: Before the harvesting robotic arm enters the contact risk segment of the candidate approach path, the pre-contact reference state of the distal high-value organ and the connection link is collected. Based on the contact signals acquired by the harvesting robotic arm within the contact risk zone, a contact confirmation index is calculated. Contact safety boundary indicators are determined based on historical harvesting boundary tables. Compare the contact confirmation index with a preset index threshold to obtain a first comparison result, and match a pause command from a preset command table based on the first comparison result; Compare the contact safety boundary index with a preset boundary threshold to obtain a second comparison result, and match a withdrawal instruction from the preset instruction table based on the second comparison result; The harvesting robotic arm is controlled according to the pause command and the retraction command.

[0014] The apparatus provided in this disclosure, wherein the evaluation module is specifically used for: Within the preset time window, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation are calculated. Based on the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation, the credibility of observation of the remote high-value organ is calculated. If the observation confidence level is greater than or equal to the preset confidence level threshold, then a remote propagation risk assessment is performed based on the direct response to obtain a risk assessment value. If the observation confidence is less than the preset confidence threshold, and one of the force observation confidence, tactile observation confidence, and alternative observation confidence is greater than or equal to the corresponding preset threshold, then a remote propagation risk assessment is performed based on the alternative response to obtain a risk assessment value. If the observation confidence is less than a preset confidence threshold, and the confidence of force observation, tactile observation, and alternative observation are all less than their corresponding preset thresholds, then the risk indicators of the direct response and alternative response are recalculated using conservative risk adjustment. If the risk indicator is within the preset risk range, the adjusted risk indicator value is output. If the risk indicator is outside the preset risk range, then the contact risk segment is replaced.

[0015] The apparatus provided in this disclosure, wherein the evaluation module is specifically used for: The corresponding remote response vector is determined based on the direct response or alternative response. The corresponding response baseline quantity is matched from the standard range table based on the remote response vector. The standardized response vector of the corresponding remote high-value organ and the connection link is calculated based on the response baseline quantity and the remote response quantity. The risk assessment value is calculated based on the standardized response vector.

[0016] The apparatus provided in this disclosure, wherein the comparison module is specifically used for; The comparison results are obtained by comparing the assessment results with the preset risk acceptance level; If the comparison result is that the assessment result is greater than the risk tolerance, then the contact risk section is replaced; if the comparison result is that the assessment result is less than or equal to the risk tolerance, then the harvesting robotic arm is controlled to pass through the contact risk section.

[0017] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the multimodal sensing contact risk response method provided in this disclosure.

[0018] This disclosure also provides a computer-readable storage medium storing a computer program for executing the multimodal sensing contact risk response method provided in this disclosure.

[0019] The technical solution provided in this disclosure has the following advantages compared with the prior art: The multimodal sensing contact risk response method provided in this disclosure determines the location of distant high-value organs and their connecting links based on the target mature fruit location. It determines candidate approach paths and their contact risk segments based on the target mature fruit location and the posture of the harvesting robotic arm. It obtains the pre-contact reference state of the distant high-value organs and their connecting links corresponding to the contact risk segment. It executes pause and retraction commands in response to contact signals within the contact risk segment. Within a preset time window, it acquires direct and alternative responses to assess the remote propagation risk, compares the assessment results with risk acceptance, and determines the execution result for the contact risk segment. This method can determine whether the path can continue when slight contact or a contact trend occurs in the contact risk segment based on the distant response, reducing the risk of damage to immature fruits, buds, flower buds, tender shoots, and key branching nodes. It also reduces the risk of misjudgment when observations are unreliable, minimizes retraction and path searching, and improves harvesting efficiency and safety. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 A flowchart illustrating the multimodal sensing contact risk response method provided in this embodiment of the disclosure; Figure 2 A schematic diagram of the structure of the multimodal sensing contact risk response device provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0024] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] To address the aforementioned issues, this disclosure provides a multimodal sensing contact risk response method, which will be described below with reference to specific embodiments.

[0029] Figure 1 This is a flowchart illustrating a multimodal sensing contact risk response method provided in an embodiment of the present disclosure. The method can be executed by a multimodal sensing contact risk response device, which can be implemented in software and / or hardware and is generally integrated into an electronic device.

[0030] Example 1: This embodiment of the present disclosure provides a multimodal sensing contact risk response method, comprising: S101: Obtain the target mature fruit site, determine the connection link between the distal high-value organ site and the distal high-value organ site within a first preset range based on the target mature fruit site, and determine the candidate approach path based on the target mature fruit site and the posture of the picking robot arm, wherein the candidate approach path includes a contact risk segment. S102: Based on the contact risk segment, obtain the pre-contact reference state between the distal high-value organ and the connection link; in response to the contact signal obtained by the picking robot arm within the contact risk segment, control the picking robot arm to execute a preset instruction according to the contact signal; the preset instruction includes a pause instruction and a retraction instruction. S103: Within a preset time window determined according to the preset instruction, obtain the direct response of the remote high-value organ and the alternative response of the connection link, perform a remote propagation risk assessment based on the direct response and the alternative response, and obtain an assessment result, the assessment result including a risk assessment value and an adjusted risk index value. S104: Compare the assessment results with the preset risk tolerance, and determine the execution result of the exposure risk segment based on the comparison results.

[0031] In this embodiment, the target mature fruit site is the fruit's spatial coordinates, serving as the endpoint of path planning and the center of the first preset range expansion, used to determine the search area for distant high-value organs. The first preset range is the boundary of the spatial region extending outward from the target mature fruit site, used to limit the identification range of distant high-value organ sites. The distant high-value organ site is the coordinate determined by matching feature points from the type feature table, serving as the endpoint of the connecting link, used to quantify the target endpoint of contact propagation. The connecting link is the line connecting the distant high-value organ site and the target mature fruit site, used to identify the spatial reference of the contact risk segment. The contact risk segment is the path interval in the candidate approach path where the distance to the connecting link is less than a preset distance condition, used to trigger pre-contact reference state acquisition and contact signal detection. The candidate approach path is the trajectory with the highest path score from the picking robot arm pose to the target mature fruit site, used to execute the approach motion. The pre-contact reference state is the initial position, attitude, and mechanical data of the distant high-value organ and the connecting link acquired by visual sensors, force sensors, and tactile sensors before entering the contact risk segment, used to provide a benchmark for comparing deviations before and after contact. The contact signal is a physical contact electrical or force signal detected by the reaping robot arm within the contact risk zone via contact sensors or force sensors. It serves as the condition for triggering pause and retraction commands. The execution process involves the reaping robot arm control system receiving a contact signal and, according to a preset logical sequence, activating a pause command to stop forward movement and then activating a retraction command to reverse direction. This provides a tiered response control for unexpected contact events. The pause command is a control command that immediately stops the reaping robot arm's current forward movement and keeps it stationary at its current position, preventing further increase in contact force and potential damage to distal high-value organs. The retraction command is a retraction command that moves the reaping robot arm away from the contact position along its original approach path, removing it from the high-risk contact area. The preset time window is a fixed-length period of time starting from the execution of the pause or retraction command, used to limit the data acquisition cycle for direct and alternative responses. The acquisition process involves continuously acquiring physical response data of the distal high-value organ and its connecting links within the preset time window using a vision camera, force sensor, and tactile sensor, capturing dynamic changes after contact. Direct response refers to raw response data of distal high-value organ positional displacement and morphological changes directly observed by visual sensors within a preset time window. This data serves as the primary basis for assessing the direct impact of contact on distal high-value organs. Alternative response refers to indirect response data of tension and strain transmission within the connection link detected by force or tactile sensors within a preset time window. This data provides an alternative assessment basis when direct observation is unreliable. Distant propagation risk assessment calculates the probability and severity of damage or displacement caused by the mechanical disturbance generated by the contact signal propagating along the connection link to the distal high-value organ based on direct or alternative response data. This quantifies the safety consequences of the contact event.The risk assessment value is a quantified value calculated using a remote propagation risk assessment algorithm. It represents the expected severity of damage to high-value organs caused by the current contact event and serves as a benchmark for comparison with risk acceptance. The adjusted risk index value is a risk index value recalculated using a conservative risk adjustment method when the reliability of visual observation, force observation, tactile observation, and alternative observation is insufficient. It provides a safer alternative assessment value when sensor data is unreliable. The comparison process involves comparing the risk assessment value or adjusted risk index value with a preset risk acceptance level to determine whether the current contact risk is within an acceptable range, deciding whether to change the contact risk segment or proceed through it. The risk acceptance level is a pre-set risk threshold parameter representing the maximum acceptable risk level allowed by the system, providing a judgment boundary for the comparison process. The contact risk segment execution result is the final action decision determined based on the comparison results, including changing or proceeding through the contact risk segment, guiding the next motion control of the harvesting robotic arm to achieve safe harvesting.

[0032] In the blueberry picking example, the target ripe fruit location is the blueberry fruit coordinates (50, 80, 30). The first preset range is a spherical area with a radius of 30mm centered on the blueberry fruit. The distal high-value organ location is the distal end of the blueberry stem at coordinates (52, 78, 32). The connecting link is the line connecting the distal end of the blueberry stem to the fruit. The contact risk segment is the path interval in the candidate approach path that is less than 2mm away from the blueberry stem. The candidate approach path is the robotic arm's trajectory with the highest path score. The pre-contact reference state is the initial position (52, 78, 32)mm and posture of the blueberry stem recorded before the robotic arm enters. The contact signal is the 0.3N force signal detected by the force sensor when the robotic arm touches the blueberry stem. The execution process is to first activate a pause command to stop moving forward, and then activate a retraction command to move backward 5mm. The pause command keeps the robotic arm stationary at the blueberry stem contact position. The retraction command moves the robotic arm away from the blueberry stem along the original path. The preset time window is 2 seconds after the pause. The acquisition process involves collecting blueberry stem response data within 2 seconds using visual and force sensors. The direct response is visually observed as a 1mm offset in the blueberry fruit and a 2° bend in the stem. The alternative response is force-detected as a 0.2N increase in internal tension within the blueberry stem. The distal propagation risk assessment calculates a 0.3 probability of fruit drop due to the contact force propagating along the stem. The risk assessment value is 0.3. The adjusted risk index value is a conservatively calculated 0.5 when observations are insufficient. The comparison process compares 0.3 with a risk acceptance rate of 0.4. The risk acceptance rate is 0.4. The contact risk segment execution result is 0.3, which is less than 0.4; therefore, the contact risk segment is passed.

[0033] The working principle and beneficial effects of this embodiment are as follows: Based on the target mature fruit site, the distal high-value organ site and its connecting links are determined; based on the target mature fruit site and the posture of the harvesting robotic arm, candidate approach paths and their contact risk segments are determined; the pre-contact reference state of the distal high-value organ and its connecting links corresponding to the contact risk segment is obtained; pause and retraction commands are executed in response to contact signals within the contact risk segment; direct and alternative responses are obtained within a preset time window to assess the distal propagation risk; the assessment results are compared with the risk acceptance rate to determine the execution result of the contact risk segment. This allows for the determination of whether the path can continue based on the distal response when slight contact or a contact trend occurs within the contact risk segment, reducing the risk of damage to immature fruits, buds, flower buds, tender shoots, and key branching nodes; reducing the risk of misjudgment when observations are unreliable; reducing retraction and path searching; and improving harvesting efficiency and safety.

[0034] Example 2: The method provided in this embodiment of the present disclosure obtains a target mature fruit site, determines the connection link between a distal high-value organ site within a first preset range and the distal high-value organ site based on the target mature fruit site, and determines candidate approach paths based on the target mature fruit site and the pose of the harvesting robotic arm, including: Obtain point cloud information of the target mature fruit, and determine the location of the target mature fruit based on the point cloud information; Based on the target mature fruit locus, expand outward to a first preset range, identify a preset high-value organ type within the first preset range, determine the feature points of the high-value organ type from the type feature table, and determine the distal high-value organ locus based on the feature points. The connection link of the distant high-value organ site is determined based on the distant high-value organ site and the target mature fruit site; Starting from the pose of the picking robotic arm and ending at the target ripe fruit location, at least one initial approach path is determined. Based on preset distance conditions, the contact risk segment in each initial approach path is determined. The path score of the corresponding initial approach path is calculated based on the contact risk segment. The initial approach path with the highest score among the path scores is selected as the candidate approach path.

[0035] In this embodiment, point cloud information is a set of three-dimensional coordinate data of the target ripe fruit, used to determine the target ripe fruit site. High-value organ type is a preset important biological structure category, and corresponding feature point extraction rules are matched from the type feature table. Feature points are geometrically representative coordinate positions on high-value organs, used to locate distal high-value organ sites. The picking robot arm pose is the current position and attitude of the robot arm's end effector, serving as the starting point for path planning. The initial approach path consists of multiple candidate motion trajectories generated from the picking robot arm pose to the target ripe fruit site, providing a basis for path selection. The preset distance condition is a rule used to determine the distance threshold between points on the path and connecting links, used to identify contact risk segments. The path score is a quantitative score calculated based on parameters such as the length and position of the contact risk segment, used to compare the merits of different initial approach paths. The corresponding calculation process is as follows: ,in, This represents the j-th candidate approach path; express The above parameter is the path point of s; s represents the path parameter, s=0 is the starting point, s=1 is the ending point; ,in, Representing path points to the set The minimum distance, if it satisfies The corresponding road segment will then be marked as a contact risk segment; This represents a set of distant high-value organ sites and their corresponding connectivity links; The contact risk distance threshold, ,in, This represents the comprehensive risk value of the l-th contact risk segment in the j-th candidate path; This represents the set of distant high-value organs or connection links associated with the l-th contact risk segment in the j-th candidate path; This represents the adjusted risk index value of an individual in the k-th distant high-value organ or connection link. ,in, This represents the prior risk index value of the k-th remote high-value organ or connection link; ,in, This indicates the observational reliability of the k-th distant high-value organ or its connecting link; Indicates the reliability of alternative observations; Indicates the weighting coefficient of the alternative observation; Indicates the reliability of tactile observations; Indicates the weighting coefficient of tactile observation; Indicates the reliability of force perception observation; This represents the force perception observation weighting coefficient; This represents the reliability of visual observations of the k-th distant high-value organ or connection link. ,in, Indicates the confidence level of the detection; Indicates the occlusion ratio; Indicates depth or point cloud completeness; Indicates the stability of continuous frame tracking; Indicates the visual observation weighting coefficient; express The k-th remote high-value organ or connection link; if If the path segment is deemed safe, you can continue approaching; if If the path segment is identified as a warning, speed reduction, stiffness reduction, or path switching will be implemented. If the path segment is deemed critical, a rollback or abort will be executed. Indicates the safety threshold; Indicates the warning threshold; Indicates the critical threshold; . ,in, Indicates the j-th candidate path The overall path score is calculated, with a smaller value indicating a better path. This represents the remote transmission risk weighting coefficient, with a value range of [0.5, 0.8]. This represents the path length weighting coefficient; This represents the total length of the j-th candidate approach path; This indicates a reference value for the path length. This represents the execution time weighting coefficient; This represents the estimated execution time value for the j-th candidate proximity path; This indicates a reference value for execution time. This represents the weighting coefficient for the posture changes of the robotic arm used for harvesting. This represents the change in the robotic arm's posture for the j-th candidate approach path.

[0036] Based on the aforementioned blueberry picking example, the point cloud information is a set of three-dimensional coordinate points collected from the surface of the blueberry fruit, such as (50.2, 80.1, 30.5), (50.5, 79.8, 30.6), and (49.8, 80.3, 30.4), used to determine the blueberry fruit location (50, 80, 30). The high-value organ type is the blueberry stalk, which is extracted using cylindrical skeleton extraction rules matched from the type feature table. The feature point is the geometrically representative distal endpoint coordinate (52, 78, 32) on the blueberry stalk, used to locate the distal stalk location. The picking robot arm pose is the current position (200, 150, 100) and orientation (0°, 30°, 0°) of the end effector, serving as the starting point for path planning. The initial approach paths consist of three candidate motion trajectories generated from the robotic arm pose to the blueberry fruit location (50, 80, 30). Path A passes through (180, 140, 110), Path B passes through (170, 130, 105), and Path C passes through (190, 145, 115), providing a basis for path selection. A preset distance condition is set at a threshold of 3mm between the point on the path and the connecting link to the blueberry stem, used to identify contact risk segments. The path score is a quantitative score calculated based on the length of the contact risk segment: Path A has a contact risk segment length of 5mm and scores 85 points; Path B has a contact risk segment length of 2mm and scores 95 points; Path C has a contact risk segment length of 8mm and scores 75 points, used to compare the merits of different initial approach paths.

[0037] The working principle and beneficial effects of this embodiment are as follows: The target mature fruit point cloud information is obtained to determine the target mature fruit location; a first preset range is expanded outward to identify high-value organ types; feature points and distal high-value organ locations are determined from a type feature table; a connection link is determined based on the distal high-value organ location and the target mature fruit location; an initial approach path is determined with the picking robot arm pose as the starting point and the target location as the ending point; contact risk segments are determined based on preset distance conditions, and path scores are calculated; the path with the highest score is selected as the candidate approach path. By locating the target fruit and distal high-value organs through point cloud positioning, constructing connection links, and evaluating contact risk segments in the approach path, path scoring and optimization are achieved, reducing the risk of accidental contact with distal high-value organs during picking and improving the safety of picking path planning.

[0038] Example 3: The method provided in this embodiment of the present disclosure obtains a pre-contact reference state between the distal high-value organ and the connection link based on the contact risk segment, responds to a contact signal obtained by the harvesting robot arm within the contact risk segment, and controls the harvesting robot arm to execute a preset command according to the contact signal, including: Before the harvesting robotic arm enters the contact risk segment of the candidate approach path, the pre-contact reference state of the distal high-value organ and the connection link is collected. Based on the contact signals acquired by the harvesting robotic arm within the contact risk zone, a contact confirmation index is calculated. Contact safety boundary indicators are determined based on historical harvesting boundary tables. Compare the contact confirmation index with a preset index threshold to obtain a first comparison result, and match a pause command from a preset command table based on the first comparison result; Compare the contact safety boundary index with a preset boundary threshold to obtain a second comparison result, and match a withdrawal instruction from the preset instruction table based on the second comparison result; The harvesting robotic arm is controlled according to the pause command and the retraction command.

[0039] In this embodiment, the contact confirmation index is a quantified parameter calculated based on the contact signal value, representing the intensity or confidence level of the contact event. It is used to compare with a preset index threshold to match a pause command. The contact safety boundary index is a reference value obtained from the historical harvesting boundary table, representing the limit of contact parameters allowed for safe operation. It is used to compare with the preset boundary threshold to match a withdrawal command. The historical harvesting boundary table is a database storing safety boundary parameters recorded in past harvesting operations, containing allowable limit values ​​corresponding to different types of contact events, providing a basis for querying the contact safety boundary index. The preset index threshold is a pre-set standard value for judging the contact confirmation index, serving as a reference benchmark for the first comparison process, used to determine whether to trigger the matching condition for a pause command. The preset command table is a mapping table storing control commands corresponding to different comparison results, containing matching rules for pause commands and withdrawal commands, used to output corresponding control commands based on the first and second comparison results. The preset boundary threshold is a pre-set standard value for judging the contact safety boundary index, serving as a reference benchmark for the second comparison process, used to determine whether to trigger the matching condition for a withdrawal command.

[0040] In this embodiment, ,in, Indicates contact confirmation indicators; Indicates the peak contact force; Indicates the contact confirmation force threshold; Indicates the increment of wrist force / torque; This indicates the threshold for confirming wrist torque. Indicates the increment of joint torque; This indicates the threshold for confirming joint torque. This indicates the change in joint current; Indicates the current confirmation threshold; Indicates compliant structural deformation; This indicates the deformation confirmation threshold. ,in, Indicates the contact safety boundary index; Indicates the maximum permissible contact force; Indicates the maximum permissible increment of joint torque; Indicates the maximum permissible increment of wrist torque; This represents the maximum permissible compliant structural deformation. When the contact safety boundary index is greater than or equal to 1, the robotic arm is immediately retracted or its current path is terminated. When the end effector, gripper edge, or contact head has a compliant structure, the equivalent contact force can be estimated from the compliant structural deformation, and the corresponding calculation formula is expressed as follows: ,in, Indicates the equivalent stiffness of a compliant structure; This indicates an estimated contact force; Indicates the compression or bending deformation of a compliant structure; Indicates equivalent damping; This represents the rate of change of compliant structural deformation.

[0041] Based on the aforementioned blueberry picking example, the confidence value of a force signal with a contact confirmation index of 0.3N is 0.6. The contact safety boundary index, obtained from the blueberry stem entry in the historical picking boundary table, is 0.25N. The preset index threshold is 0.5. The preset boundary threshold is 0.2N. In the preset instruction table, a first comparison result greater than the threshold matches a pause instruction, and a second comparison result greater than the threshold matches a revert instruction.

[0042] The working principle and beneficial effects of this embodiment are as follows: Before entering the contact risk zone, a pre-contact reference state of the distal high-value organ and the connecting link is collected. A contact confirmation index is calculated based on the contact signal acquired within the contact risk zone. A contact safety boundary index is determined based on a historical harvesting boundary table. A pause command is matched by comparing the contact confirmation index with a preset index threshold. A retraction command is matched by comparing the contact safety boundary index with a preset boundary threshold. The harvesting robotic arm is controlled according to the pause and retraction commands. By using both contact signals and the historical boundary table for dual judgment, pause and retraction commands are matched separately, achieving hierarchical safety control, avoiding misjudgment based on a single threshold, reducing the risk of damage to the distal high-value organ, and improving the reliability of the robotic arm's contact response.

[0043] Example 4: The method provided in this embodiment of the present disclosure acquires the direct response of the remote high-value organ and the alternative response of the connection link within a preset time window determined according to the preset instruction. Based on the direct response and the alternative response, a remote propagation risk assessment is performed to obtain an assessment result, which includes a risk assessment value and an adjusted risk indicator value. The preset time window is activated simultaneously with the execution of the preset instruction, including: Within the preset time window, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation are calculated. Based on the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation, the credibility of observation of the remote high-value organ is calculated. If the observation confidence level is greater than or equal to the preset confidence level threshold, then a remote propagation risk assessment is performed based on the direct response to obtain a risk assessment value. If the observation confidence is less than the preset confidence threshold, and one of the force observation confidence, tactile observation confidence, and alternative observation confidence is greater than or equal to the corresponding preset threshold, then a remote propagation risk assessment is performed based on the alternative response to obtain a risk assessment value. If the observation confidence is less than a preset confidence threshold, and the confidence of force observation, tactile observation, and alternative observation are all less than their corresponding preset thresholds, then the risk indicators of the direct response and alternative response are recalculated using conservative risk adjustment. If the risk indicator is within the preset risk range, the adjusted risk indicator value is output. If the risk indicator is outside the preset risk range, then the contact risk segment is replaced.

[0044] In this embodiment, visual observation confidence is a quantified confidence value obtained by a visual sensor when acquiring direct response data. It represents visual observation quality such as image sharpness and target visibility, and is used to participate in the observation confidence calculation and determine whether to use the direct response for risk assessment. Force observation confidence is a quantified confidence value obtained by a force sensor when acquiring response data. It represents force observation quality such as force signal-to-noise ratio and sensor accuracy, and is used to determine whether an alternative response can be used for assessment when the observation confidence is insufficient. Tactile observation confidence is a quantified confidence value obtained by a tactile sensor when acquiring response data. It represents tactile observation quality such as contact force distribution perception and tactile array response, and is used to determine whether an alternative response can be used for assessment when the observation confidence is insufficient. Alternative observation confidence is a comprehensive quantified confidence value obtained by a force sensor or tactile sensor when acquiring alternative response data. It represents the overall reliability of the alternative observation channel and is used to participate in the observation confidence calculation and independently determine whether an alternative response can be used for assessment. Observational confidence is an overall confidence value calculated by comprehensively considering visual observation confidence, force observation confidence, tactile observation confidence, and surrogate observation confidence. It represents the overall reliability of current observations of high-value distant organs and is used to determine whether to use a direct response, surrogate response, or conservative risk adjustment for distant propagation risk assessment. The preset confidence threshold is a pre-defined standard value for judging observational confidence, serving as a boundary condition for branch decisions and determining whether the observational confidence is sufficient to support a direct response assessment. Conservative risk adjustment is a risk indicator recalculation method used when all observation channels are unreliable. It applies a safety-biased correction to both direct and surrogate responses, outputting a more conservative adjusted risk indicator value when observational data is unreliable. The preset risk range is a pre-defined acceptable range for risk indicators, including a lower and upper threshold. It is used to determine whether the conservatively risk-adjusted risk indicator is within a safe range to decide whether to output the adjusted risk indicator value or change the contact risk segment.

[0045] In this embodiment, ,in, Indicates the risk assessment value; The weighting coefficients represent the standardized torque response in the q-direction; The weighting coefficients represent the standardized torque response in the w direction; The weighting coefficients represent the normalized response time. Weighting coefficients representing the displacement-normalized response; Weighting coefficients representing the standardized force response; Weighting coefficients representing the standardized response of the attitude angle; Weighting coefficients represent the standardized response of the deformation; all weighting coefficients are summed to equal 1. This represents the standardized displacement response of the k-th distal high-value organ or connection link. This represents the standardized angular displacement, attitude change, or bending angle response of the k-th distal high-value organ or connecting link. This represents the standardized contact force response. This represents the standardized wrist torque increment response. This represents the standardized joint torque increment response. This represents the dynamic recovery assistance response quantity for the k-th remote high-value organ or connection link; Let represent the standardized compliant structural deformation, where the numerator of all expressions is the standard quantity of the corresponding quantity. This indicates the angular displacement or bending angle change of twigs, fruit stalks, or connecting links; k represents the index of a distal high-value organ or its connecting link. This indicates the displacement of visible immature fruits, branching nodes, fruit clusters, or connecting links; Represents the remote response vector; Compliant structural deformations; This indicates optional dynamic recovery auxiliary indicators, including at least one of the following: whether the baseline range before contact is returned, the residual offset at the end of a preset time window, or the equivalent recovery time. Indicates the increment of joint torque; Indicates the increment of wrist torque; This indicates the peak contact force.

[0046] Based on the aforementioned blueberry picking example, the calculated confidence levels are as follows: visual observation confidence level is 0.7, force observation confidence level is 0.8, tactile observation confidence level is 0.6, and substitute observation confidence level is 0.75. The overall confidence level is 0.72. The preset confidence level threshold is 0.8. If the observation confidence level is less than the threshold but the force observation confidence level (0.8) is greater than the corresponding preset threshold (0.75), a substitute response is used for risk assessment. During conservative risk adjustment, the direct response offset is changed from 1mm to 2mm. The preset risk range is 0.2 to 0.8. If the risk index of 0.5 is within this range, the adjusted risk index value is output.

[0047] The working principle and beneficial effects of this embodiment are as follows: Within a preset time window, the direct response and alternative response of the connecting link from a distant high-value organ are acquired. Visual, force, tactile, alternative observation credibility, and comprehensive observation credibility are calculated. Based on a comparison of credibility with a threshold, a direct response, alternative response, or conservative risk adjustment is selected for evaluation. If the risk index is within a preset risk range, the adjusted risk index value is output; otherwise, the contact risk segment is replaced. Through multimodal observation credibility-based hierarchical decision-making, the reliability of the evaluation is ensured. When observation is insufficient, conservative adjustment or path replacement is activated to avoid risk misjudgment and improve the robustness and security of remote propagation risk assessment.

[0048] Example 5: The method provided in this embodiment of the present disclosure, wherein the remote propagation risk assessment includes: The corresponding remote response vector is determined based on the direct response or alternative response. The corresponding response baseline quantity is matched from the standard range table based on the remote response vector. The standardized response vector of the corresponding remote high-value organ and the connection link is calculated based on the response baseline quantity and the remote response quantity. The risk assessment value is calculated based on the standardized response vector.

[0049] In this embodiment, the distal response vector is a multidimensional vector constructed based on direct or alternative response data. It represents the response distribution characteristics of the distal high-value organ in spatial or mechanical dimensions and is used to match the corresponding response baseline quantity from a standard range table. The specific formula is as follows: The standard range table is a pre-established database table that stores the mapping relationship between different response vector value ranges and response baseline quantities. It is used to query the corresponding standard reference value based on the remote response vector. The response baseline quantity is a standardized reference value matched from the standard range table. It represents the standard response level of a remote high-value organ to the same stimulus under normal operating conditions and is used to calculate the standardized response vector together with the remote response quantity. The remote response quantity is the raw value obtained through actual measurement of the direct or alternative response. It represents the actual response magnitude of the remote high-value organ in the current exposure event and is used for comparison calculations with the response baseline quantity. The standardized response vector is a dimensionless vector calculated based on the response baseline quantity and the remote response quantity through ratio, difference, or normalization. It represents the normalized response characteristics after eliminating dimensions and individual differences and is used as the basic input for calculating the risk assessment value. The risk assessment value is a quantitative value calculated based on the standardized response vector through weighted summation or model calculation. It represents the expected severity of damage to the remote high-value organ caused by the exposure event and is used for subsequent calculations with risk acceptance.

[0050] Based on the aforementioned blueberry picking example, the distal response vector is (offset 1mm, bend 2°). In the standard range table, an offset of 1 to 1.5mm corresponds to a baseline of 1.2mm, and a bend of 1.5 to 2.5° corresponds to a baseline of 2°. The distal response values ​​are offset 1mm and bend 2°, and the response baseline values ​​are 1.2mm and 2°. The standardized response vector is (1 / 1.2=0.83, 2 / 2=1.0). The risk assessment value is calculated based on (0.83, 1.0) to obtain 0.3.

[0051] The working principle and beneficial effects of this embodiment are as follows: A remote response vector is determined based on the direct or alternative response; a response baseline quantity is matched; a standardized response vector is calculated; and a risk assessment value is calculated based on the standardized response vector. Standardization eliminates dimensional differences, improving the consistency and accuracy of risk assessment.

[0052] Example 6: The method provided in this embodiment compares the assessment result with a preset risk tolerance level, and determines the execution result of the exposure risk segment based on the comparison result. The execution result includes replacing the contact risk segment and passing through the contact risk segment, including: The comparison results are obtained by comparing the assessment results with the preset risk acceptance level; If the comparison result is that the assessment result is greater than the risk tolerance, then the contact risk section is replaced; if the comparison result is that the assessment result is less than or equal to the risk tolerance, then the harvesting robotic arm is controlled to pass through the contact risk section.

[0053] In this embodiment, The working principle and beneficial effects of this embodiment are as follows: The evaluation result is compared with the risk acceptance level; if the evaluation result is greater than the risk acceptance level, the contact risk segment is changed; if it is less than or equal to the risk acceptance level, the path is passed. Decisions are made based on the risk acceptance threshold to ensure safe passage; if the threshold is exceeded, the path is switched to avoid risk.

[0054] To achieve the above embodiments, this disclosure also proposes a multimodal sensing contact risk response device.

[0055] Figure 2 This is a schematic diagram of the structure of a multimodal sensing contact risk response device provided in an embodiment of this disclosure. The device 200 can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 2 As shown, the device 200 includes: a first acquisition module 201, a second acquisition module 202, an evaluation module 203, and a comparison module 204, wherein, Module 201: The first acquisition module 201 is used to acquire the target mature fruit site, determine the connection link between the distal high-value organ site and the distal high-value organ site within a first preset range based on the target mature fruit site, and determine the candidate approach path based on the target mature fruit site and the posture of the picking robot arm. The candidate approach path includes a contact risk segment. The second acquisition module 202 is used to acquire the pre-contact reference state of the distal high-value organ and the connection link based on the contact risk segment, and in response to the contact signal acquired by the picking robot arm in the contact risk segment, control the picking robot arm to execute preset instructions according to the contact signal, the preset instructions including a pause instruction and a retraction instruction. The evaluation module 203 is used to acquire the direct response of the remote high-value organ and the alternative response of the connection link within a preset time window determined according to the preset instruction, and to perform a remote propagation risk assessment based on the direct response and the alternative response to obtain an evaluation result, the evaluation result including a risk assessment value and an adjusted risk indicator value. The comparison module 204 is used to compare the evaluation result with the preset risk acceptance level, and determine the execution result of the exposure risk segment based on the comparison result.

[0056] The apparatus provided in this disclosure embodiment, wherein the first acquisition module 201 is specifically used for: Obtain point cloud information of the target mature fruit, and determine the location of the target mature fruit based on the point cloud information; Based on the target mature fruit locus, expand outward to a first preset range, identify a preset high-value organ type within the first preset range, determine the feature points of the high-value organ type from the type feature table, and determine the distal high-value organ locus based on the feature points. The connection link of the distant high-value organ site is determined based on the distant high-value organ site and the target mature fruit site; Starting from the pose of the picking robotic arm and ending at the target ripe fruit location, at least one initial approach path is determined. Based on preset distance conditions, the contact risk segment in each initial approach path is determined. The path score of the corresponding initial approach path is calculated based on the contact risk segment. The initial approach path with the highest score among the path scores is selected as the candidate approach path.

[0057] The apparatus provided in this embodiment of the present disclosure, wherein the second acquisition module 202 is specifically used for: Before the harvesting robotic arm enters the contact risk segment of the candidate approach path, the pre-contact reference state of the distal high-value organ and the connection link is collected. Based on the contact signals acquired by the harvesting robotic arm within the contact risk zone, a contact confirmation index is calculated. Contact safety boundary indicators are determined based on historical harvesting boundary tables. Compare the contact confirmation index with a preset index threshold to obtain a first comparison result, and match a pause command from a preset command table based on the first comparison result; Compare the contact safety boundary index with a preset boundary threshold to obtain a second comparison result, and match a withdrawal instruction from the preset instruction table based on the second comparison result; The harvesting robotic arm is controlled according to the pause command and the retraction command.

[0058] The apparatus provided in this disclosure, wherein the evaluation module 203 is specifically used for: Within the preset time window, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation are calculated. Based on the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation, the credibility of observation of the remote high-value organ is calculated. If the observation confidence level is greater than or equal to the preset confidence level threshold, then a remote propagation risk assessment is performed based on the direct response to obtain a risk assessment value. If the observation confidence is less than the preset confidence threshold, and one of the force observation confidence, tactile observation confidence, and alternative observation confidence is greater than or equal to the corresponding preset threshold, then a remote propagation risk assessment is performed based on the alternative response to obtain a risk assessment value. If the observation confidence is less than a preset confidence threshold, and the confidence of force observation, tactile observation, and alternative observation are all less than their corresponding preset thresholds, then the risk indicators of the direct response and alternative response are recalculated using conservative risk adjustment. If the risk indicator is within the preset risk range, the adjusted risk indicator value is output. If the risk indicator is outside the preset risk range, then the contact risk segment is replaced.

[0059] The apparatus provided in this disclosure, wherein the evaluation module 203 is specifically used for: The corresponding remote response vector is determined based on the direct response or alternative response. The corresponding response baseline quantity is matched from the standard range table based on the remote response vector. The standardized response vector of the corresponding remote high-value organ and the connection link is calculated based on the response baseline quantity and the remote response quantity. The risk assessment value is calculated based on the standardized response vector.

[0060] The apparatus provided in this disclosure embodiment includes a comparison module 204, which is specifically used for: The comparison results are obtained by comparing the assessment results with the preset risk acceptance level; If the comparison result is that the assessment result is greater than the risk tolerance, then the contact risk section is replaced; if the comparison result is that the assessment result is less than or equal to the risk tolerance, then the harvesting robotic arm is controlled to pass through the contact risk section.

[0061] The multimodal sensing contact risk response device provided in this disclosure can execute the multimodal sensing contact risk response method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0062] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the multimodal sensing contact risk response method described above.

[0063] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0064] The following is a detailed reference. Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this disclosure. The electronic device in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0065] like Figure 3 As shown, the electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 302 or a program loaded from memory 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device. The processor 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0066] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0067] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a memory 308, or installed from a ROM 302. When the computer program is executed by the processor 301, it performs the functions defined in the multimodal sensing contact risk response method of embodiments of this disclosure.

[0068] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0069] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0070] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0071] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the aforementioned multimodal sensing contact risk response method.

[0072] Electronic devices can be programmed with computer program code in one or more programming languages ​​or combinations thereof to perform the operations of this disclosure. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0074] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0075] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0076] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0077] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0078] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0079] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A multimodal sensing contact risk response method, characterized in that, include: Obtain the target mature fruit site, determine the connection link between the target mature fruit site and the distal high-value organ site within a first preset range, and determine the candidate approach path based on the target mature fruit site and the posture of the picking robot arm. The candidate approach path includes a contact risk segment. Based on the contact risk segment, the pre-contact reference state of the distal high-value organ and the connection link is obtained. In response to the contact signal obtained by the picking robot arm within the contact risk segment, the picking robot arm is controlled to execute preset instructions according to the contact signal. The preset instructions include a pause instruction and a retraction instruction. Within a preset time window determined according to the preset instruction, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, a remote propagation risk assessment is performed to obtain the assessment result, which includes a risk assessment value and an adjusted risk indicator value. The assessment results are compared with the preset risk tolerance, and the execution result of the exposure risk segment is determined based on the comparison results.

2. The method according to claim 1, characterized in that, The process involves: acquiring the target mature fruit locus; determining the connection link between the target mature fruit locus and a distal high-value organ locus within a first preset range; and determining candidate approach paths based on the target mature fruit locus and the pose of the harvesting robotic arm, including: Obtain point cloud information of the target mature fruit, and determine the location of the target mature fruit based on the point cloud information; Based on the target mature fruit locus, expand outward to a first preset range, identify a preset high-value organ type within the first preset range, determine the feature points of the high-value organ type from the type feature table, and determine the distal high-value organ locus based on the feature points. The connection link of the distant high-value organ site is determined based on the distant high-value organ site and the target mature fruit site; Starting from the pose of the picking robotic arm and ending at the target ripe fruit location, at least one initial approach path is determined. Based on preset distance conditions, the contact risk segment in each initial approach path is determined. The path score of the corresponding initial approach path is calculated based on the contact risk segment. The initial approach path with the highest score among the path scores is selected as the candidate approach path.

3. The method according to claim 1, characterized in that, Based on the contact risk segment, a pre-contact reference state of the distal high-value organ and the connection link is obtained. In response to a contact signal acquired by the harvesting robotic arm within the contact risk segment, the harvesting robotic arm is controlled to execute preset instructions according to the contact signal, including: Before the harvesting robotic arm enters the contact risk segment of the candidate approach path, the pre-contact reference state of the distal high-value organ and the connection link is collected. Based on the contact signals acquired by the harvesting robotic arm within the contact risk zone, a contact confirmation index is calculated. Contact safety boundary indicators are determined based on historical harvesting boundary tables. Compare the contact confirmation index with a preset index threshold to obtain a first comparison result, and match a pause command from a preset command table based on the first comparison result; Compare the contact safety boundary index with a preset boundary threshold to obtain a second comparison result, and match a withdrawal instruction from the preset instruction table based on the second comparison result; The harvesting robotic arm is controlled according to the pause command and the retraction command.

4. The method according to claim 1, characterized in that, Within a preset time window determined according to the preset instructions, the direct response of the remote high-value organ and the alternative response of the connection link are acquired. Based on the direct response and the alternative response, a remote propagation risk assessment is performed to obtain an assessment result, which includes a risk assessment value and an adjusted risk indicator value. The preset time window is activated simultaneously with the execution of the preset instruction, including: Within the preset time window, the direct response of the remote high-value organ and the alternative response of the connection link are obtained. Based on the direct response and the alternative response, the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation are calculated. Based on the credibility of visual observation, the credibility of force observation, the credibility of tactile observation and the credibility of alternative observation, the credibility of observation of the remote high-value organ is calculated. If the observation confidence level is greater than or equal to the preset confidence level threshold, then a remote propagation risk assessment is performed based on the direct response to obtain a risk assessment value. If the observation confidence is less than the preset confidence threshold, and one of the force observation confidence, tactile observation confidence, and alternative observation confidence is greater than or equal to the corresponding preset threshold, then a remote propagation risk assessment is performed based on the alternative response to obtain a risk assessment value. If the observation confidence is less than a preset confidence threshold, and the confidence of force observation, tactile observation, and alternative observation are all less than their corresponding preset thresholds, then the risk indicators of the direct response and alternative response are recalculated using conservative risk adjustment. If the risk indicator is within the preset risk range, the adjusted risk indicator value is output. If the risk indicator is outside the preset risk range, then the contact risk segment is replaced.

5. The method according to claim 4, characterized in that, The remote transmission risk assessment includes: The corresponding remote response vector is determined based on the direct response or alternative response. The corresponding response baseline quantity is matched from the standard range table based on the remote response vector. The standardized response vector of the corresponding remote high-value organ and the connection link is calculated based on the response baseline quantity and the remote response quantity. The risk assessment value is calculated based on the standardized response vector.

6. The method according to claim 1, characterized in that, The assessment results are compared with the preset risk tolerance level, and the execution result of the exposure risk segment is determined based on the comparison results. The execution result includes replacing the contact risk segment and passing through the contact risk segment, including: The comparison results are obtained by comparing the assessment results with the preset risk acceptance level; If the comparison result is that the assessment result is greater than the risk tolerance, then the contact risk section is replaced; if the comparison result is that the assessment result is less than or equal to the risk tolerance, then the harvesting robotic arm is controlled to pass through the contact risk section.

7. A multimodal sensing contact risk response device, the device comprising: The first acquisition module is used to acquire the target mature fruit site, determine the connection link between the target mature fruit site and the distal high-value organ site within a first preset range based on the target mature fruit site, and determine the candidate approach path based on the target mature fruit site and the posture of the picking robot arm. The candidate approach path includes a contact risk segment. The second acquisition module is used to acquire the pre-contact reference state of the distal high-value organ and the connection link based on the contact risk segment, and in response to the contact signal acquired by the picking robot arm in the contact risk segment, control the picking robot arm to execute preset instructions according to the contact signal, the preset instructions including a pause instruction and a retraction instruction; The assessment module is used to acquire the direct response of the remote high-value organ and the alternative response of the connection link within a preset time window determined according to the preset instruction, and to conduct a remote propagation risk assessment based on the direct response and the alternative response to obtain an assessment result, which includes a risk assessment value and an adjusted risk indicator value. The comparison module is used to compare the assessment results with a preset risk tolerance level, and determine the execution result of the exposure risk segment based on the comparison results.

8. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

Images