Spraying task generation and execution method, system and device based on interactive prompts

Abstract

The application discloses a spraying task generation and execution method, system and device based on interactive prompts, and comprises the following steps: obtaining task entry data containing interactive prompt data or external task indication data, determining an explicit task area or an implicit effective work range as a candidate task expression based on the task entry data, the boundary of the task area being constrained by the interactive prompt data, establishing a task association between the candidate task expression and the physical surface of the current physical target to obtain a final task expression, performing release verification on the final task expression, and generating robot executable task data to control the robot work after the verification is passed. The application realizes reliable conversion of artificial interactive intention to robot executable task, reduces the risk of out-of-bound mis-spraying, and improves the task release reliability and flexibility.

Description

Technical Field

[0001] This invention relates to the field of robotic automated spraying technology, specifically to a method, system, and device for generating and executing spraying tasks based on interactive prompts. Background Technology

[0002] In non-target painting scenarios such as automotive repair and touch-up, to balance human experience with the robot's automated execution capabilities, the painting task area is typically defined by displaying a vehicle image on a human-machine interface and having the operator delineate the area to be repaired. However, existing solutions often directly convert the pixel coordinates delineated by the operator on the screen into robot work coordinates, or simply rely on a single image segmentation algorithm to extend the manual trajectory. This approach results in excessive coupling between human input and the final task boundary, making it difficult to adapt to accuracy fluctuations under low-cost hardware. Furthermore, due to the lack of a reliable correlation verification mechanism from virtual interaction to the physical surface, accidental spraying outside the designated area is highly likely when the vehicle's position changes, the vehicle model is incompatible, or a prohibited painting area exists. In addition, existing technologies struggle to accommodate various low-cost implementation paths such as pre-stored model area selection, cross-device distributed operation, and trajectory generation without independent area objects, limiting the system's flexibility and security. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, and device for generating and executing spraying tasks based on interactive prompts. This invention achieves a reliable conversion of human interactive intentions into robot-executable tasks, reduces the risk of accidental spraying outside designated areas, and improves the reliability and flexibility of task issuance.

[0004] The technical solution of the present invention: a method for generating and executing spraying tasks based on interactive prompts, comprising:

[0005] Obtain task entry data, which includes interactive prompt data generated based on target representation data or external task indication data; wherein, the external task indication data needs to be parsed to determine at least two of its data source, target or component identity, coordinate system or relative reference before it can enter subsequent processing;

[0006] Candidate task expressions are determined based on the task entry data, and the candidate task expressions include explicit task areas or implicit effective work ranges; wherein, when the task entry data includes the interactive prompt data, the boundaries of the candidate task expressions are constrained by the interactive prompt data;

[0007] The candidate task representation is associated with the physical surface of the current physical target to obtain the final task representation;

[0008] The final task expression is published and validated. After the validation passes, robot-executable task data is generated to control the robot to perform operations on the current physical target.

[0009] The interactive prompt data in the above method includes at least one of positive prompt data, negative prompt data, or stop boundary data; the interactive prompt data is formed from one or more, continuous or segmented, same or different interactive terminal inputs, and is combined before or during the determination of the candidate task expression;

[0010] Based on the task entry data, candidate task representations are determined, including:

[0011] Based on the target representation data and interactive prompt data, perform region generation processing to generate an explicit task region;

[0012] Specifically, when the interactive prompt data includes positive prompt data, the explicit task area includes the physical location corresponding to the positive prompt data; when the interactive prompt data includes negative prompt data, the explicit task area excludes the physical location corresponding to the negative prompt data; when the interactive prompt data includes stop boundary data, the boundary of the explicit task area is limited by the stop boundary data.

[0013] The aforementioned method, which determines candidate task representations based on the task entry data, includes:

[0014] Generate at least one of the following: spraying centerline, coverage unit, or spraying trajectory based on the task entry data;

[0015] The union of the spray pattern footprints corresponding to the spray centerline, the covering unit, or the spray trajectory is determined as the implicit effective working range;

[0016] The release verification includes: when the task entry data includes interactive prompt data, verifying whether the implicit valid job range meets the preset prompt consistency condition; and verifying whether the implicit valid job range meets the prohibited job condition.

[0017] In the aforementioned method, the target representation data includes pre-stored target representation data;

[0018] Associating the candidate task representation with the physical surface of the current physical target includes:

[0019] Identify external entity markers set on the current physical target or its associated environment, or identify the natural features of the current physical target;

[0020] Based on the external entity markers or natural features, the candidate task representations in the pre-stored target representation data are mapped to the physical surface of the current physical target;

[0021] The final task expression's job semantics and final job scope are determined based on at least one of the following: screen task selection instructions, external task semantics, verified preset rules, automatic confirmation results, or semi-automatic confirmation results.

[0022] The aforementioned method, wherein the external entity marking includes at least one of the following: special color tape, reflective material, coded label, printed mark, sticker, magnetic component, calibration plate, positioning fixture or stop;

[0023] The external entity markers are used to enclose, approximately enclose, or segment the repair area of ​​the current physical target, or to indicate the pose of the current physical target.

[0024] The external entity markers are identified by a vision device, and the machine candidate boundaries or marker relative coordinates are extracted.

[0025] The process of determining the job semantics and final job scope of the final task expression includes:

[0026] The machine candidate boundary or its corresponding result on the pre-stored target representation data is displayed on the human-computer interaction interface;

[0027] The screen task selection instruction received by the operator, which selects between task type, core area to be repaired, transition area, prohibited area, or candidate area, is used to form the final task expression.

[0028] The aforementioned method performs release verification on the final task expression, including:

[0029] Verify whether the correlation between the target representation data, candidate task expression, and physical surface belongs to a compatible target version;

[0030] Verify whether the work scope corresponding to the final task expression intrudes into the prohibited work area;

[0031] Verify the serviceability of the robot in expressing the final task;

[0032] When the release verification passes, the robot automatically releases the task data and controls the robot to perform the operation without receiving additional manual confirmation instructions.

[0033] The aforementioned method, wherein the target representation data includes initial display data;

[0034] Based on the target representation data and interactive prompt data, a region generation process is performed to generate an explicit task region, including:

[0035] A rough work area is determined based on the initial display data and interactive prompt data;

[0036] Determine the supplementary acquisition field of view or sensor pose based on the rough working area;

[0037] Control the vision module or robotic arm to move to the supplementary acquisition field of view or sensor pose to acquire supplementary acquisition data;

[0038] Based on the supplementary collected data, the region generation process is re-executed to obtain the explicit task region;

[0039] The boundaries of the regenerated explicit task region must not exceed the permitted physical range corresponding to the rough task region.

[0040] The aforementioned method, wherein the robot-executable task data includes at least one of the following positional relationship expressions:

[0041] A homogeneous transformation matrix containing Cartesian coordinate components and tool orientation;

[0042] Surface parameter coordinates and distance along the surface normal;

[0043] Mesh patch identifiers and in-plane coordinates;

[0044] The local coordinates of the component or the relative coordinates of the marker;

[0045] Robot joint target;

[0046] Image spatial control increment based on real-time image feedback;

[0047] The robot's executable task data is generated and sent in a streaming manner by region block, trajectory segment, or image feedback during the robot's operation.

[0048] A spray painting task generation and execution system based on interactive prompts, comprising:

[0049] The task entry acquisition module is used to acquire task entry data, which includes interactive prompt data generated based on target representation data or external task indication data.

[0050] The task generation module is used to determine candidate task expressions based on the task entry data. The candidate task expressions include explicit task regions or implicit effective work ranges. When the task entry data includes interactive prompt data, the boundaries of the candidate task expressions are constrained by the interactive prompt data.

[0051] The task association module is used to establish a task association between the candidate task expression and the physical surface of the current physical target to obtain the final task expression;

[0052] The task publishing module is used to perform publishing verification on the final task expression;

[0053] The execution control module is used to generate robot-executable task data after the verification is passed, so as to control the robot to perform operations on the current physical target.

[0054] A control device, comprising:

[0055] One or more processors;

[0056] At least one memory stores instructions that can be executed by the one or more processors;

[0057] Robot task output interface;

[0058] When the instruction is executed by the one or more processors, the control device performs the aforementioned interactive prompt-based spraying task generation and execution method, and sends the robot-executable task data to the robot through the robot task output interface.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. This invention uses interactive prompt data as a constraint rather than a direct boundary, combined with region generation processing or spray footprint union calculation, so that the generation process of candidate task expression can integrate image features, geometric continuity or process parameters, thereby correcting the arbitrariness and inaccuracy of human operation while retaining human intent, and improving the robustness of task region generation.

[0061] 2. This invention establishes a task association between candidate task expressions and the physical surface of the current physical target, and on this basis, performs release verification including version compatibility, prohibited work area intrusion checks, and robot serviceability checks, constructing a closed-loop verification mechanism from virtual interaction to physical execution. This mechanism can intercept erroneous task data caused by vehicle model mismatch, coordinate offset, or environmental changes before the robot takes action, effectively reducing the risk of accidental spraying due to out-of-bounds errors, and supports automatic execution after the release verification passes, reducing unnecessary manual secondary confirmation steps.

[0062] 3. This invention supports a technical approach that selects areas based on pre-stored target representation data and performs on-site delayed association using external entity markers or natural features, decoupling task planning time from physical operation time. This allows operators to complete high-quality task editing using a standard model before the vehicle arrives, and high-precision registration can be completed on-site only through markers or feature recognition, significantly improving operation preparation efficiency and the system's adaptability to unstructured environments.

[0063] 4. This invention is compatible with both explicit task regions and implicit effective working ranges, and supports multiple positional relationship expression methods and streaming data transmission. This design eliminates the need for the system to rely on independent 3D region objects or complete coordinate point sets, enabling it to adapt to various underlying architectures from lightweight trajectory generation to complex online visual feedback control, thus enhancing the technical solution's compatibility with different hardware configurations and control strategies. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the spraying task generation and execution method based on interactive prompts according to an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram illustrating the constraint relationship between interactive prompt data and candidate task expressions in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the process of partial supplementary data collection and task generation after coarse selection in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram illustrating the association path between candidate task representations and physical target surfaces in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the delayed association process between the pre-stored target representation and the external entity tag in an embodiment of the present invention;

[0069] Figure 6 This is a schematic diagram illustrating cross-device interaction and the generation of implicit effective work range according to an embodiment of the present invention;

[0070] Figure 7 This is a schematic diagram showing the layout of the core area to be repaired, the transition area, and the prohibited spraying area according to an embodiment of the present invention;

[0071] Figure 8 This is a schematic diagram of the task release verification and automatic execution process according to an embodiment of the present invention. Detailed Implementation

[0072] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0073] Before introducing specific embodiments, the core concepts and data relationships involved in this invention will be explained.

[0074] Regarding the separation of interactive input and final task representation: Operator's interactive input on-site or pre-stored target representation data can directly determine the candidate range of the model surface or form machine-readable cue constraints. The system then performs model surface attachment, physical target association, region or coverage generation, trajectory planning, or online control, and obtains the final task representation after publication and verification. This invention does not require that manual input, candidate task representation, and final task representation have the same boundaries, nor does it require saving the direct conversion results from manual input to robot coordinates point by point. However, it requires that manual input has at least one verifiable direct selection, inclusion, exclusion, position limitation, boundary limitation, candidate selection, confirmation, or correction relationship with the final task representation. For example, if an operator draws a wide stroke in the middle of a damaged area with their finger, this stroke only constitutes a positive cue. The image segmentation algorithm extends along color or texture differences to the actual damage edge, and the resulting candidate area is significantly larger than the input stroke. As long as the candidate area contains positive cue, does not enter negative cue or prohibited areas, and passes physical surface association and publication verification, the final task representation can be formed. For example, the system first automatically identifies multiple suspected damage areas and displays them on the screen. The operator clicks on one of the candidate areas or presses the confirmation button. At this time, the interactive input does not contain any boundary coordinates, but it has the function of selecting or confirming the candidate area, which still falls under the interactive prompt constraint involved in this invention. Another example is that before the vehicle arrives, the operator precisely paints or selects a set of mesh patches on a pre-stored 3D vehicle model displayed on a tablet computer. This set of patches can directly become the candidate task range in the model's coordinate domain. After the vehicle arrives, the system determines the correspondence between the model and the actual vehicle through natural features, special colored tape, or other external objects, and then generates coverage, trajectory, or online control targets.

[0075] Interactive prompt data can be formed by one or more selection operations, consecutively or in segments, on the same or different interactive terminals. Each selection fragment can be combined into a point set, stroke set, mask, boundary, component set, or equivalent task data at the entry device, task generation node, or task association node before entering physical surface association and publication verification. Regardless of whether the combination occurs before or after importation, as long as each selection fragment collectively generates verifiable constraints on the scope or semantics of the final task expression, it belongs to the interactive prompt data referred to in this invention.

[0076] Regarding the cue constraint field: Cue constraints can be represented as discrete points, binary or multi-valued masks, distance fields, weight fields, semantic labels, primitive sets, component identifiers, or rule sets. Positive cues require the corresponding location to be a candidate for the task, while negative cues require the corresponding location to be excluded, stopping the expansion of the boundary restriction region. Tolerance constraints limit the maximum deviation of the candidate boundary from the manual coarse selection. Candidate confirmation is used to select one from multiple candidate results. In one optional mathematical expression, a cue value P(x) is constructed for a location x on the target surface or in the image domain. P(x) is positive near positive cues, negative near negative cues, and zero or low weight for uncued locations. The candidate evaluation S(x) is formed by combining image or geometric consistency A(x), boundary cost G(x), and template consistency T(x). The candidate region can be determined by the threshold of S(x), connected components, minimum cut, level set, or the output of a validated model. This expression is only used to illustrate the constraint relationship of cue on region generation and does not limit the invention to a specific formula or machine learning model. To prevent the algorithm from infinitely expanding the coarse selection area to adjacent panels, the system can set hard stop boundaries and soft stop boundaries. Windows, lights, trim strips, holes, and physical obstructions are typically used as hard stop boundaries; panel seams, abrupt changes in curvature, or color changes can be used as hard stop or high-cost soft stop boundaries.

[0077] Regarding the automatic region generation method: Automatic region generation is not limited to deep learning. Low-cost implementations can utilize region growing, graph cutting, contour finding, edge snapping, color space clustering, interactive foreground segmentation, or morphological processing from open-source computer vision libraries; semantic segmentation, instance segmentation, cue-driven segmentation, or multimodal models can also be used. Regardless of the algorithm type, this invention can be applied as long as the interactive input is used to constrain, select, confirm, or correct the automatically generated final task expression. Candidate regions can be generated directly from the currently displayed image, or from original high-resolution images, depth data, point clouds, computer-aided design models, component templates, or supplementary scan data that are task-related to the displayed image. The data displayed on the screen does not need to have the same resolution or data format as the data processed by the algorithm. For the component template method, the operator can click on components such as the left front door, fender, or bumper. The system calls the corresponding template based on the current vehicle model, component recognition results, or geometric registration, and allows further narrowing of the template range based on the operator's local smearing. For external marker prompts, the system can detect special color tape, reflective tape, coded labels, stickers, magnetic components, calibration plates, positioning clamps, blocks, projected patterns, light spots, or other external objects, and extract their identity, position, direction, edges, mutual distance, or spatial arrangement. Markers can limit the expansion range of the area, or form candidate results such as areas to be repaired, center lines, transition areas, or prohibited areas. After the operator selects, confirms, excludes, or corrects the candidate results on the screen, the system determines the candidate task expression.

[0078] Regarding the generation of coverage without independent region objects: Candidate task representations do not need to be saved as region objects with independent names or fixed data structures. The operator can draw a centerline, several guide points, or rough strokes on the screen. The system automatically generates coverage units, spraying trajectories, or continuously updated image spatial control targets based on the spray width, spray gun posture, trajectory spacing, overlap rate, and allowable deviation. In this case, the union of the spray footprints, spray contribution range, or online control range corresponding to each trajectory position is taken as the implicit effective working range. The system uses this implicit effective working range to check whether positive prompts are covered, whether negative prompts and prohibited spraying areas are excluded, whether boundary deviations are within tolerance, and whether the robot can provide service. Therefore, even if the implementer claims that there are no areas to be repaired, candidate masks, or 3D task objects within the system, as long as the machine-generated centerline, trajectory, coverage unit, or online control target forms an effective working range constrained by interactive prompts on the physical surface, and automatic spraying is performed accordingly, it still falls within the task generation technology chain involved in this invention.

[0079] Regarding the physical surface association method: When the area to be repaired can be approximated as a plane or a segmented plane, a regular fixed camera can be used for calibration. Through homography matrix, calibration mesh interpolation, or local segmentation transformation, the display area is associated with the two-dimensional physical coordinates of the vehicle surface. This is then combined with the known spray gun working distance or real-time ranging to determine the tool pose. For areas with gentle curvature, such as car doors, hoods, or bumpers, the target surface can be divided into several small patches. Local mapping is used within each patch, and continuity correction is performed between patches. If a location exceeds the calibration effective domain or the projection error exceeds a threshold, an automatic painting task is not issued. When depth values ​​and camera intrinsic parameters are obtained, the display pixels can be back-projected into camera coordinate points based on the camera imaging model, and then transformed to common reference coordinates or robot base coordinates based on the camera pose. If the pixels do not have direct depth, the camera rays can intersect with a calibrated vehicle surface, point cloud, or computer-aided design model. When using component models or digital twins, manual selection can be performed on the model first, and then the current vehicle image, point cloud, reproducible natural features, or external markers can be registered with the model to migrate the candidate regions on the model to the current physical vehicle. In another implementation, the final task representation remains in the current camera image space, and the system continuously generates robot control increments or joint targets based on the error between the current position of the spray gun or end effector in the image and the target coverage position.

[0080] Generation of the core area to be repaired and the transition area. The final task area may include the core area to be repaired, the transition area between old and new coatings, the feathering area, the varnish covering area, and the prohibited spraying area. The operator may first apply the core area to be repaired, then delineate the transition area; or only apply the core area, and the system automatically generates the transition area according to the curved surface distance, the edge of the plate part and the spraying process; or the system automatically generates the two areas first, and the operator confirms them at one time. In one embodiment, the geodesic distance d(x) relative to the boundary of the core area is calculated on the curved surface of the vehicle, the region where 0≤d(x)≤d1 is set as the color paint expansion region, the region where d1≤d(x)<d2 is set as the feathering or fusion region, and the regions are deducted from the window, lamp, trim strip and other prohibited areas. d1 and d2 may be fixed values, or may vary with the curvature, spray width direction, plate edge, coating type, and the transition boundary additionally delineated by the operator. The core area to be repaired and the transition area may be adjacent, partially overlapped, contained or nested, and both carry regional semantics and process attributes respectively to support different speeds, flow rates, spraying distances, overlapping rates, spray gun inclination angles, trigger advance amounts or spraying passes.

[0081] Embodiment 1: This embodiment provides a spraying task generation and execution method based on interactive prompts. The method processes task inputs from different sources through a unified process, and establishes a closed-loop verification mechanism from virtual interaction to physical execution. Specifically, as Figure 1 shown in and Figure 5 shown in, the method includes the following steps S101 to S106.

[0082] Step S101: Obtain task entry data. Task entry data includes interactive prompt data generated based on target representation data, or external task instruction data. The external task instruction data must be parsed to determine at least two of the following: data source, target or component identity, and coordinate system or relative reference, before proceeding to subsequent processing. In this embodiment, interactive prompt data is typically generated by the display and input module, such as operator actions like smearing, selecting, clicking to confirm, or correcting on a vehicle image on a touchscreen. These actions are converted into machine-readable prompt constraints. External task instruction data can originate from a maintenance work order system, offline programming software, damage detection equipment, or upstream manufacturing execution system. Its form can be a file or interface message containing coordinates, area identifiers, or trajectory parameters. Upon receiving external task instruction data, the system first performs parsing processing: confirming the identity of the target vehicle or component corresponding to the data, the coordinate system or relative reference upon which the data is based, and whether the data version matches the current system. For example, the coordinates of the damaged area output by the damage detection equipment must simultaneously carry the component identifier (e.g., "left front door") and the coordinate reference (e.g., "component local coordinate system" or "sensor coordinate system"). The system uses this information to determine whether the data can establish a valid correspondence with the current physical target. If the correspondence can be verified, the parsed data is used as the basis for candidate task expression; if it cannot be verified, at least one of the following is used to re-establish the association: the currently observed image, natural features, external entity markers, vehicle or component reference, or fixture reference. External data that has not undergone the above parsing and association is not directly used for subsequent task generation simply because it already contains coordinate or trajectory information. It should be understood that regardless of whether the data originates from human screen interaction or external automated equipment, the system treats it as a unified task entry point and does not change the subsequent task generation and verification logic due to different sources. This design allows the system to be compatible with both human experience-driven maintenance scenarios and data-driven automated production line scenarios, improving the method's versatility.

[0083] Step S102: Based on the task entry data, candidate task expressions are determined. These expressions include explicit task regions or implicit effective work areas. When the task entry data includes interactive prompt data, the boundaries of the candidate task expressions are constrained by this data. In this step, the explicit task region refers to a two-dimensional mask, three-dimensional mesh set, or parametric surface interval with clear geometric boundaries. The implicit effective work area refers to the actual work coverage area determined by the union of spray footprints corresponding to the spray centerline, coverage unit, or spray trajectory. In this case, the system may not store independent region objects. The term "constraint" means that the interactive prompt data (such as finger smearing trajectory) is not directly equivalent to the final task boundary, but rather serves as the basis for the algorithm to generate or correct candidate task expressions. For example, the operator's smearing only serves as a positive prompt seed; the region generation algorithm automatically expands to form the final boundary by combining image texture or geometric features. Alternatively, the operator may simply click to confirm an automatically identified candidate area; this confirmation action constitutes a constraint on the candidate range. This approach improves the tolerance for human input while ensuring that the generated results meet process requirements.

[0084] Step S103: Establish a task association between the candidate task representation and the physical surface of the current physical target to obtain the final task representation. For example... Figure 4 As shown, the essence of this step is to establish a mapping relationship between candidate representations in virtual space and the target surface to be painted in physical space. This mapping can be achieved through methods such as camera calibration parameters, depth back projection, model registration, natural feature matching, or external marker recognition. For example, when the candidate task representation is generated on a pre-stored 3D model, the system needs to calculate the transformation matrix between the model coordinate system and the current vehicle physical coordinate system to migrate the virtual region to the actual vehicle surface; when the candidate task representation is generated on a 2D image, the system needs to project it onto the 3D physical surface through ray intersection or homography transformation. Only through task association can the virtual interactive intent be transformed into a physically meaningful work instruction. This step is a crucial step in preventing erroneous operations due to vehicle model mismatch, positional offset, or viewpoint differences.

[0085] Step S104 involves performing a release verification on the final task representation. Release verification is a prerequisite for automated robot execution, not an optional auxiliary function. The verification includes at least: verifying whether the target representation data, candidate task representations, and their association with the physical surface are compatible versions to prevent older tasks from being executed on newer vehicle bodies; checking whether the final work area encroaches on prohibited work zones (such as windows, lights, trim strips, etc.); and evaluating the serviceability of the painting robot for the current task (such as accessibility, posture singularities, spray distance compliance, etc.). The verification status is only considered passed when all preset safety and process conditions are met. This mechanism constructs a safety defense line from digital planning to physical execution, effectively reducing the risks of collisions and accidental spraying in automated operations.

[0086] Step S105: After successful verification, generate robot-executable task data. In this step, based on the verified final task expression and combined with spraying process parameters (such as spray width, flow rate, speed, overlap rate, etc.), generate control instructions that the spraying robot can recognize and execute. This data can be represented as a homogeneous transformation matrix sequence, joint angle targets, surface parameter paths, or image space control increments, etc. It should be understood that the generation process can be an offline complete planning or an online streaming generation, as long as the data content is sufficient to drive the robot to complete the specified task.

[0087] Step S106: Control the robot to perform tasks on the current physical target. In this step, the generated robot-executable task data is sent to the controller of the painting robot, driving the robotic arm and spray gun to perform painting operations on the target according to the predetermined trajectory and process parameters. In this embodiment, since the previous steps have completed strict release verification, the robot can start working directly after receiving the execution command, without the need for operators to perform additional task release confirmation for the same final task expression, thereby improving work efficiency; personnel protection, emergency stop, explosion protection, and robot safety functions are still executed by independent safety-related control chains.

[0088] Through steps S101 to S106 above, this embodiment constructs a complete upper-level method closed loop. This closed loop establishes two core pillars: "interactive prompt constraints" and "physical association verification." The former decouples the rigid binding between manual input and machine execution, allowing for various low-cost and highly flexible interaction methods; the latter ensures the accuracy and security of mapping virtual tasks to the physical world.

[0089] Preferably, the interactive prompt data includes at least one of positive prompt data, negative prompt data, or stop boundary data. Specifically, such as... Figure 2As shown, positive cue data is used to indicate the interior or core location of the area to be worked on. Its data form can be a pixel mask created by the user drawing on the screen, a seed point set generated by clicking, or an imported set of 2D primitives. Negative cue data is used to explicitly exclude areas that do not require work, such as a background mask generated by the user through an erasing operation or a designated no-spray component marker. Stop boundary data is used to limit the physical limits of area expansion, such as panel seams, window edges, lamp outlines, or boundary lines manually drawn by the user. These different types of cue data are converted into constraints that the region generation algorithm can recognize, for example, as hard foreground / background constraints or soft weights in graph cut algorithms, or as seed points and termination criteria in region growing algorithms.

[0090] Based on target representation data and interactive prompt data, region generation processing is performed to generate an explicit task region. Specifically, when the interactive prompt data includes positive prompt data, the explicit task region includes the physical location corresponding to the positive prompt data; when the interactive prompt data includes negative prompt data, the explicit task region excludes the physical location corresponding to the negative prompt data; when the interactive prompt data includes stop boundary data, the boundary of the explicit task region is limited by the stop boundary data. Further, when the interactive prompt data includes positive prompt data, the boundary of the explicit task region is allowed to not completely coincide with the input trajectory corresponding to the positive prompt data; when the interactive prompt data includes stop boundary data, the boundary of the explicit task region is allowed to coincide with the input trajectory of the stop boundary data. This feature reflects the decoupling relationship between manual input and machine-generated boundaries. For example, when an operator roughly paints in the middle of a vehicle damage area, the painting trajectory only serves as a positive prompt seed, automatically expanding outward based on the image's color gradient, texture continuity, or depth normal variation until it encounters a stop boundary or meets a preset consistency threshold to form a closed boundary. Therefore, the resulting explicit task area often closely matches the actual damage edge compared to the finger smear trajectory, and the two do not overlap geometrically. This design eliminates the need for operators to perform high-precision outlining operations, reducing interaction difficulty and time costs, while using algorithms to ensure the technological rationality of the task boundaries.

[0091] As an advanced implementation method, when the precision of the initial display data is insufficient to support accurate region generation, this invention also introduces a local supplementary acquisition mechanism after coarse selection. Specifically, such as... Figure 3As shown, the target characterization data includes initial display data; a coarse working area is determined based on the initial display data and interactive prompt data. For example, the operator selects a general area containing the scratch on the panoramic image of the vehicle; this area is the coarse working area. Subsequently, the system determines the supplementary acquisition field of view or sensor pose based on the coarse working area. A preset edge margin (e.g., 10mm to 100mm) is added to the coarse working area to ensure that the damage boundary falls completely within the subsequent high-precision field of view, and the movement path and target pose of the camera or robotic arm end effector are planned accordingly.

[0092] The vision module or robotic arm is moved to a supplementary acquisition field of view or sensor pose to acquire supplementary acquisition data. During this process, the painting robot or independent mobile vehicle drives a high-resolution camera or depth sensor to a designated position to acquire local detail images or point cloud data. Then, based on the supplementary acquisition data, the region generation process is re-executed to obtain the explicit task region. Because the supplementary acquisition data has higher spatial resolution or signal-to-noise ratio, the regenerated region boundary can more accurately fit minor damage or coating transition zones. Crucially, the boundary of the regenerated explicit task region must not exceed the permissible physical range corresponding to the coarse work area. This constraint ensures that even if the high-precision algorithm discovers new texture features locally, its expansion behavior is still limited by the initial manual coarse selection intention, preventing the task region from drifting to adjacent panels or prohibited painting areas due to algorithm oversensitivity. Through this two-level processing mechanism of "coarse selection of the region and fine selection of the edge," this embodiment maintains the macro-control of the final task range by interactive prompts while ensuring operational accuracy, further enhancing the robustness and safety of the system in complex unstructured environments.

[0093] As another embodiment, the process of determining candidate task representations based on task entry data can adopt a different technical path than the aforementioned embodiments, namely, not generating or saving independent explicit region objects. Specifically, at least one of spraying centerline, coverage unit, or spraying trajectory is generated based on task entry data. Unlike the method of generating closed masks through image segmentation or region growing in the aforementioned embodiments, this embodiment directly converts the operator's interactive input (such as a drawn centerline, a specified guide point, or a rough stroke) into path data or coverage sequence with process attributes. For example, if the operator draws an open curve along the direction of the area to be repaired on the display and input module, the system does not attempt to close the curve into a region, but instead uses it as the spraying centerline, and automatically generates a series of continuous spraying trajectory segments or discrete coverage units by combining preset spray gun height, fan width, and overlap rate parameters. This processing method is significantly different from traditional region objects in terms of data structure. The system memory only stores trajectory parameters, joint targets, or image space control increments, and there is no two-dimensional mask or three-dimensional mesh set named "area to be repaired", thereby reducing data processing overhead and adapting to certain specific robot control architectures.

[0094] Furthermore, the union of the spray footprints corresponding to the spray centerline, coverage unit, or spray trajectory is defined as the implicit effective working area. Here, the "implicit effective working area" is not a pre-stored geometric entity, but a logical concept dynamically defined by the trajectory execution effect. Specifically, the spray footprint refers to the surface projection area where paint is actually deposited or effectively applied under a specific tool posture for a single coverage unit or trajectory point; the union is the geometric superposition of the spray footprints corresponding to all planned trajectory points on the physical surface. It should be understood that although the system does not explicitly construct the boundary representation of this union internally, mathematically and physically, this union is equivalent to a defined working coverage area. This definition ensures that even if the implementer completely omits the creation and storage steps of the area object in the software implementation, this technical feature still holds true as long as the generated trajectory sequence forms a controlled coverage set on the physical surface.

[0095] For this type of non-explicit task expression, the release verification logic is adjusted accordingly. Release verification includes: when the task entry data includes interactive prompt data, verifying whether the implicit valid work area meets the preset prompt consistency condition; and verifying whether the implicit valid work area meets the prohibited work condition. Since there is no area mask that can be directly used for Boolean operations, a verification strategy based on sampling or parsing calculation is adopted. For the prompt consistency condition, the system calculates the coverage rate of the implicit valid work area (i.e., the union of spray footprints) to the corresponding position of the positive prompt data. If the coverage rate is lower than a preset threshold (e.g., 95%), the verification is deemed to have failed; at the same time, it checks whether the union intersects with the physical position corresponding to the negative prompt data. If there is an intersection, the prompt consistency condition is deemed not met. For the prohibited work condition, the system projects the boundary of the trajectory sampling point or coverage unit onto the vehicle model to verify whether it crosses hard stop boundaries such as windows, lights, and trim strips. Figure 6 As shown, in cross-device interaction scenarios, external interaction devices may only transmit centerline or trajectory files without transmitting the original selected area image. In this case, the task generation device reconstructs the implicit effective operation range based on the received trajectory data and performs the above verification, which not only ensures lightweight data transmission under the distributed architecture, but also maintains strict control over the safety boundary of the operation.

[0096] Preferably, the target representation data includes pre-stored target representation data. Unlike the previous embodiments that relied on real-time on-site image or point cloud data for interaction, this embodiment employs a spatiotemporally decoupled task generation mode. Specifically, the operator can load pre-stored target representations on the display and input module for task planning before the vehicle enters the painting station, or even before the vehicle arrives at the repair shop. The pre-stored target representations can be 3D CAD models, 2D unfolded diagrams, component schematics, or parametric surface data of standard vehicle models, and do not depend on the real-time state of the current physical target. This design allows the task editing process to break free from the rhythm constraints of the production site, allowing operators to utilize high-precision models in a more comfortable environment for refined area selection and process pre-setting, significantly improving work preparation efficiency and planning quality.

[0097] The process of establishing a task association between candidate task representations and the physical surface of the current physical target is achieved by identifying external entity markers set on the current physical target or its associated environment, or by identifying the natural features of the current physical target. When the vehicle is in position, the absolute coordinates on the pre-stored model are not directly used; instead, it is treated as a virtual template to be registered. External entity markers act as reference anchors connecting the virtual and physical spaces in this process. For example, external entity markers may include at least one of the following: specially colored tape, reflective materials, coded labels, printed marks, stickers, magnetic attachments, calibration plates, positioning clamps, or stops. These markers are temporarily placed or attached to the vehicle surface, obstructions, ground references, or positioning clamps, and their physical form has characteristics that are easily recognizable by machine vision. It should be understood that although this embodiment lists various specific forms, in practical applications, any external object that can be stably detected by sensors and has a clear spatial orientation can serve as an external entity marker, without being limited to specific colors, materials, or coding rules. External entity markers can be used to enclose, approximately enclose, or segmentally indicate the area to be repaired of the current physical target, or to indicate the pose of the current physical target. For example, the operator can use specially colored tape to create a rectangular frame around the scratch, or simply apply reflective stickers to the four corners of the damaged area; magnetic attachments can also be used to define the origin of the component's coordinate system. This flexibility allows the system to adapt to various field conditions, from precise boundary definitions to coarse pose cues.

[0098] Based on external entity markers or natural features, candidate task representations in the pre-stored target representation are mapped to the physical surface of the current physical target. Specifically, the vision device identifies external entity markers and extracts machine candidate boundaries or marker relative coordinates. Based on the identified marker positions, orientations, and geometric relationships with corresponding reference points in the pre-stored model, a transformation matrix is ​​calculated from the pre-stored model coordinate system to the current physical target coordinate system. Subsequently, the system projects the candidate task representations (such as mesh patch sets, texture coordinate ranges, or parametric surface paths) previously selected on the pre-stored target representation onto the actual vehicle surface using this transformation matrix. If natural features such as tires, wheel arches, and headlights are simultaneously identified, the system can also fuse multi-source information for joint optimization registration to improve mapping accuracy. This process achieves a closed-loop technology of "offline area selection and online binding," effectively solving the problem of cumbersome interaction on the actual vehicle in traditional solutions.

[0099] It should be noted that after completing the external entity labeling or natural feature recognition and mapping, the system determines the final task's semantics and scope based on at least one of the following: screen task selection instructions, external task semantics, verified preset rules, automatic confirmation results, or semi-automatic confirmation results. Screen task selection is one important implementation method: such as... Figure 7As shown, the system displays candidate boundaries of the machine or their corresponding results on pre-stored target representations, and receives screen task selection instructions from the operator regarding task type, core area to be repaired, transition area, prohibited area, or candidate area, to form the final task expression. When the category, encoding, or deployment rules of external entity markers are pre-bound to the task type, core area to be repaired, transition area, or prohibited area, and the automatic identification results meet the preset reliability and consistency thresholds, the system can also automatically determine the job semantics and job scope according to verified preset rules; when the conditions are only partially met, the system can output candidate results for semi-automatic confirmation by the operator. Regardless of the method used, the final task expression must pass the target version compatibility, prohibited operation conditions, and robot serviceability release verification before generating robot-executable task data.

[0100] As a preferred approach, the process of performing release verification on the final task expression is constructed as a multi-dimensional security defense to ensure the accuracy and safety of robot operations. For example... Figure 8 As shown, the following key verification dimensions must be completed sequentially or in parallel before generating data for robot-executable tasks.

[0101] First, the system verifies whether the relationship between the target representation data, candidate task expressions, and physical surfaces belongs to compatible target versions. Specifically, the system checks whether the currently loaded vehicle model version number, component identifier, coordinate system definition version, and external entity marker configuration version are consistent with the basic data on which the final task expression was generated. For example, if the operator uses a 3D model of vehicle model A when selecting an area on the pre-saved target representation, but the actual model on site is a modified model B, or the vehicle's parking position causes a change in the fixture reference, the version compatibility check will fail. This mechanism effectively prevents task misalignment caused by vehicle model iteration, component replacement, or environmental reference drift, avoiding mapping incorrect virtual planning onto mismatched physical entities.

[0102] Secondly, the system verifies whether the work area corresponding to the final task description intrudes into the prohibited work area. The final determined core area to be repaired, the transition area between old and new coatings, and the implicit effective work area are spatially compared with the preset prohibited painting areas. Prohibited painting areas typically include vehicle windows, lamp surfaces, rubber seals, air intake grilles, and other non-painting sensitive components. The verification logic not only checks whether the area boundaries directly overlap but also reserves a safe buffer distance based on the diffusion characteristics of the painting process. When an intersection between the work area and the safe buffer distance is detected, the system determines it as an intrusion into the prohibited work area. This check has the highest priority; even if the operator mistakenly selects an area containing vehicle lights, the system will forcibly intercept it, thereby reducing the risk of damage to expensive components due to human error.

[0103] Next, the serviceability of the robot for the final task is verified. This step aims to verify whether the painting robot can complete the planned task with high quality under the current physical constraints. Specifically, this includes: reachability analysis, verifying that all target points are within the effective working envelope of the robotic arm; posture feasibility analysis, checking whether the spray gun will trigger joint limits, singularities, or self-collision with the vehicle body during its movement along the trajectory; and process parameter compliance analysis, confirming that the spray gun normal angle, nozzle-to-surface distance (spray distance), and moving speed are within the paint deposition process window. If a certain trajectory segment is geometrically reachable, but the spray distance exceeds the allowable tolerance, resulting in uneven coating thickness, the serviceability verification will also be marked as an anomaly. Through this physical-level pre-verification, the system ensures that the generated task data is not only mathematically correct but also engineering-executable.

[0104] Based on the results of the above multi-dimensional verification, one of three states is output: Pass, Condition Allowed, or Prohibited. When all necessary check items meet the preset thresholds, the state is "Pass"; when some non-critical indicators (such as slightly larger local spray distance but still within the acceptable range) are not optimal but basic quality can still be guaranteed, the state may be "Condition Allowed". In this case, the system may automatically adjust process parameters (such as reducing the moving speed) to adapt to the deviation; and when any critical safety item (such as version mismatch, intrusion into the no-spray zone, or unreachable) fails, the state is "Prohibited".

[0105] Specifically, under the stated permissible conditions, the system automatically completes the process parameter adjustment, which is considered a successful verification. Without receiving additional manual confirmation, it automatically releases the robot's executable task data and controls the robot to perform the operation. This is significantly different from the "planning-confirmation-execution" three-stage process commonly used in existing technologies. In this invention, because the pre-verification stage already covers comprehensive verification of version consistency, spatial security, and physical feasibility, the system has a high degree of confidence in the correctness of the task data. Therefore, once the verification status is "passed" or "conditions permitted," the painting robot is immediately started, without the operator needing to click the "confirm" button again or perform a secondary enable on the teach pendant. This design shortens the waiting time from task planning to actual operation, significantly improving overall production efficiency, especially in batch repair or assembly line operation scenarios. At the same time, this also forces the system to possess extremely high verification reliability, because the convenience of automatic execution is built upon rigorous safety logic.

[0106] Conversely, if the verification result is "prohibited," the system will never execute the program despite the error. At this point, the corresponding exception handling process is triggered based on the specific reason for the failure. For example, if it's a version mismatch, the system prompts for a rescan or selection of the correct vehicle model; if it's a partial intrusion into a no-spray zone, the system automatically trims the out-of-bounds area or requests manual boundary correction; if serviceability is insufficient, the system attempts to replan the route or prompts for adjustment of the vehicle's parking position. Only after the exception is resolved and the re-verification passes will the system enter the automatic execution phase. This strict control mechanism ensures that while pursuing high-efficiency automation, operational safety and quality are always prioritized.

[0107] Example 2: This example provides a spraying task generation and execution system based on interactive prompts. The system includes a task entry acquisition module, a task generation module, a task association module, a task publishing module, and an execution control module. These modules logically form a closed-loop processing unit, but their physical implementation is not limited to a single hardware device; rather, they can be flexibly deployed according to the needs of the actual working environment.

[0108] The task entry acquisition module is used to acquire task entry data, which includes interactive prompt data generated based on target representation data or external task instruction data. In specific implementations, this module can be jointly implemented by the target data acquisition module and the display and input module. The target data acquisition module is responsible for collecting on-site images, depth data, or calling pre-stored target representations. Its data source can be a local camera, a remote sensor, or a cloud-based model library. The display and input module provides a human-computer interaction interface, receiving interactive inputs such as scribbling, clicking, and confirming from the operator, or receiving interactive results imported from external interactive devices. It should be understood that the functional boundary of the task entry acquisition module lies in unifying heterogeneous input sources (whether real-time manual operation or offline external files) into a task entry data format that the system can recognize internally, thereby shielding the differences in data sources.

[0109] The task generation module determines candidate task representations based on task entry data. These representations include explicit task regions or implicit effective work areas. When the task entry data includes interactive cue data, the boundaries of the candidate task representations are constrained by this cue data. This module is typically implemented collaboratively by a cue constraint module and a task generation module. The cue constraint module parses the interactive input into constraints such as positive cue, negative cue, or stop boundaries. The task generation module then uses these constraints, combined with image features or geometric models, to generate candidate task representations through segmentation or trajectory planning algorithms. In distributed scenarios, the task generation module can run entirely on a standalone workstation or server, receiving only compressed or encoded cue constraint data without transmitting the original high-bandwidth video stream, thus significantly reducing reliance on network bandwidth.

[0110] The task association module is used to establish a task association between candidate task representations and the physical surface of the current physical target, resulting in the final task representation. This module is the key hub for realizing the transformation from virtual planning to physical execution. It calculates the precise pose of the candidate task representation in the current physical coordinate system by calling calibration parameters, identifying external entity markers, or matching natural features. In one implementation, the task association module can be integrated into the robot controller, directly utilizing the robot's own vision system for online registration; in another implementation, this module can also be deployed on edge computing nodes to pre-complete coarse registration between the model and the field, and then pass the transformation matrix to downstream units. Regardless of the deployment location, its core function is to ensure that the final task representation maintains a consistent spatial correspondence with the actual surface of the target to be painted.

[0111] The task release module performs release verification on the final task expression. As the system's task release control unit, this module enforces version compatibility, no-spray zone intrusion checks, and robot serviceability verification before execution control. A permission signal is generated only when all verification items pass. This module is typically tightly coupled with the execution control module, but it can also operate as a standalone process verification service. For example, in high-risk operational scenarios, the task release module can be deployed on a separate process controller or industrial computer, communicating with the robot controller via a standard industrial communication interface, enabling the task release verification logic to run independently.

[0112] The execution control module generates executable task data for the robot after successful verification, controlling the robot to perform tasks on the current physical target. This module transforms the verified final task expression into low-level instructions that the painting robot can execute, such as joint trajectories, tool poses, or image space control increments. It can directly drive robot movement or send task data to third-party robot controllers via standard interfaces (such as ROS, Modbus TCP, or proprietary APIs). In streaming execution mode, the execution control module can also dynamically adjust subsequent task data blocks based on real-time feedback, achieving adaptive control of the painting process.

[0113] It should be noted that data interaction between the above modules is not required to be completed on the same computer or within the same process. Modules can communicate via local networks, remote networks, removable storage media, offline files, or application programming interfaces (APIs). For example, in a preferred distributed application scenario, maintenance personnel use a handheld tablet (carrying the display and input modules) to select an area outside the explosion-proof zone; the tablet sends the interaction results to an industrial control computer (carrying the task generation and task association modules) via Wi-Fi; after completing area generation and physical association, the industrial control computer sends the final task expression to the robot controller (carrying the task publishing and execution control modules) via Ethernet. This distributed architecture not only adapts to the physical limitations of complex environments such as automotive repair shops but also allows for independent upgrades and maintenance of each part of the system, greatly improving the system's scalability and engineering implementation capabilities. Through this modular system design, this invention ensures that the core technical logic, from interactive prompt constraints to physical execution verification, remains complete and consistent regardless of changes in hardware topology.

[0114] Example 3: This example provides a control device, which includes one or more processors, at least one memory, and a robot task output interface. Specifically, the processor may be one or more combinations of a general-purpose central processing unit, a graphics processing unit, a field-programmable gate array, or an application-specific integrated circuit (ASIC), used to perform complex image recognition, coordinate transformation, and trajectory planning calculations. The memory stores instructions executed by the one or more processors, which logically correspond to the various steps of the interactive prompt-based spraying task generation and execution method described in the foregoing examples. The robot task output interface serves as a communication bridge between the control device and the physical execution terminal, and may take the form of an industrial Ethernet interface, a fieldbus interface, a dedicated motion control card interface, or a wireless communication module, used to send robot-executable task data to the spraying robot.

[0115] When the instructions stored in the memory are executed by one or more processors, the control device performs the interactive prompt-based painting task generation and execution method as described in any of the foregoing embodiments, and sends robot-executable task data to the painting robot through the robot task output interface. In this process, the control device does not merely perform abstract data processing, but rather directly intervenes in the real world by acquiring, converting, and outputting physical signals. For example, when performing the steps of external entity marker recognition and physical surface association, the processor analyzes the image stream acquired by the vision sensor in real time, calculates the spatial transformation matrix between the pre-stored target representation and the current physical target surface, and applies this matrix to the coordinate transformation of the candidate task expression; subsequently, the final task expression, after release verification, is converted into a sequence of control instructions containing position, attitude, and process parameters, and sent to the servo driver or main controller of the painting robot via the robot task output interface at millisecond intervals.

[0116] This tight integration of hardware architecture and instruction execution produces significant technical effects. First, by fusing and verifying multi-source sensor data in real time through the processor, mapping errors caused by vehicle parking deviations, marker installation errors, or model version inconsistencies are effectively reduced, ensuring the accuracy of virtual task migration to the physical surface. Second, in addition to transmitting the work trajectory, the robot task output interface can also carry ordinary process status feedback signals; when the release verification detects intrusion into the no-spray zone or serviceability anomalies, the processor can send a task cancellation or pause request through this interface, keeping the erroneous task prohibited from release or stopping its issuance. Personnel protection, emergency stop, explosion-proof, and robot safety functions are executed by independent safety-related control chains and are not replaced by the aforementioned robot task output interface. Furthermore, the control device's streaming processing and sending mechanism for instructions enables the painting robot to dynamically adjust its actions based on real-time feedback during execution, further improving the adaptability to complex curved surface operations and the consistency of coating quality. It should be understood that although this embodiment uses a single control device as an example, in a distributed deployment scenario, the functions of the aforementioned processor, memory, and interface can also be implemented collaboratively by multiple physical nodes, as long as their overall execution logic and technical effects are consistent with this invention.

[0117] Example 4: This example provides a comprehensive implementation method for multimodal interaction and distributed application scenarios, aiming to demonstrate the collaborative workflow of the technical features in the aforementioned examples in a real automotive repair shop environment. This scenario integrates mechanisms such as cross-device interaction, delayed association of pre-stored models, generation of implicit effective work scope, multi-dimensional release verification, and streaming task execution, verifying the flexibility and robustness of the technical solution in complex unstructured environments.

[0118] In this application scenario, the task generation and execution process specifically includes the following steps S1 to S5.

[0119] Step S1: Obtain interactive prompt data generated based on the pre-stored target representation. Specifically, before the vehicle enters the painting station, the maintenance personnel use a tablet computer, which serves as an external interactive device, to load the pre-stored target representation. The pre-stored target representation can be a 3D CAD model or a 2D unfolded drawing of a standard vehicle model, existing independently of the current physical target. The operator specifies the approximate range and intended transition zone of the area to be repaired by smearing or circling on the tablet screen. The interactive result formed by this operation includes location information and preliminary process intentions, but is not directly equivalent to the final physical work boundary. This spatiotemporal decoupling design allows task planning to break free from the rhythm constraints of the production site, enabling operators to perform fine-tuning in a more suitable environment, significantly improving work preparation efficiency.

[0120] Step S2 establishes a task association between the candidate task representation and the current physical target surface. Once the vehicle is in place, the external interactive device imports the task file containing interactive prompt data to the industrial control computer via Wi-Fi. The task association module on the industrial control computer controls the on-site industrial camera to acquire images of the vehicle surface and identifies the special-colored tape pasted around the damaged area of ​​the vehicle body as external entity markers. During this process, the system not only uses the external entity markers to calculate the spatial transformation matrix between the pre-stored target representation and the current physical target, but also displays the machine candidate boundaries generated by the marker recognition results on the human-machine interface. The operator further issues on-screen task selection instructions for the task type, the core area to be repaired, the transition area between old and new coatings, or the prohibited painting area. This instruction transforms simple geometric markings into a final task representation with clear process semantics, completing the precise mapping and semantic binding from the virtual model to the physical entity.

[0121] Step S3: Determine the implicit effective work area based on interactive prompt data. Unlike generating explicit masks, the task generation module in this embodiment directly generates a series of continuous spraying centerlines or coverage units based on imported prompt constraints and screen task selection instructions. The system determines the implicit effective work area as the geometric superposition result of the spray footprints corresponding to each spraying trajectory segment on the physical surface. Although no independent region object is constructed in the system memory, this implicit effective work area is mathematically and physically equivalent to a defined work coverage area. This processing method reduces data processing overhead and enables the system to directly perform subsequent verification and execution based on trajectory parameters, adapting to a lightweight robot control architecture.

[0122] Step S4 involves performing a release verification on the final task representation. Before generating robot-executable task data, the task release module automatically performs multi-dimensional safety checks. These checks include: verifying the compatibility of the pre-stored target representation version with the actual vehicle model; verifying whether the implicit effective working area encroaches on prohibited painting areas such as windows and lights; and verifying the reachability and posture feasibility of the painting robot to the planned trajectory. Only when all verification items meet preset thresholds is the system considered to have passed the verification. This mechanism constitutes a safety barrier from digital planning to physical execution, effectively preventing accidental painting incidents caused by vehicle model mismatch, marker displacement, or algorithm errors.

[0123] Step S5: After successful verification, the robot's executable task data is automatically generated and streamed. When the verification status is passed, the execution control module directly converts the implicit valid work area into a homogeneous transformation matrix sequence containing Cartesian coordinate components and tool posture without receiving additional manual confirmation instructions. Subsequently, this data is streamed and sent to the painting robot's controller via an industrial Ethernet interface, segment by segment. The streaming transmission mechanism avoids memory overflow or communication delays caused by loading massive amounts of trajectory points at once, while also supporting dynamic adjustment of subsequent data blocks based on real-time feedback during execution. Since the preliminary steps have completed the verification of task version, prohibited areas, and robot serviceability, the robot can start working directly when the independent safety chain conditions on site are met, without requiring manual confirmation of the same final task expression again, thereby improving the efficiency of automated continuous production.

[0124] Through steps S1 to S5 above, this embodiment fully demonstrates the complete closed-loop technology from flat panel selection, on-site marking and registration, implicit trajectory generation, automatic release verification to streaming execution. This implementation shows that the technical solution of the present invention can not only adapt to centralized processing of a single device, but also well support distributed collaboration across devices and stages; it is compatible with explicit region representation and also supports implicit control without independent region objects. This multimodal and multi-path compatibility design helps the technical solution maintain the stability and reliability of core functions when facing different hardware configurations, different process flows, and different operating habits, and helps to block low-cost implementation paths that circumvent protection by changing intermediate data structures or splitting device links.

[0125] Example 5: In this example, the system first performs damage candidate detection on the vehicle image, generating multiple candidate regions and highlighting them on the display and input module with different numbers or colors. The operator clicks on a candidate region number or clicks the confirmation button. This operation does not provide boundary coordinates but rather forms a selection and confirmation constraint for the machine's candidate regions. Figure 2As shown, although the interactive input does not contain explicit boundary information, it still has the function of selecting or confirming candidate regions and is still subject to the interactive prompt constraints involved in this invention.

[0126] If the operator removes a portion of the candidate region using an erasing pen, the erased portion forms a negative cue, and the task generation module recalculates the region boundary. If the operator adds a positive point, the algorithm adds the portion connected to that point and satisfying appearance consistency to the candidate region. Alternatively, the system can automatically generate candidate regions using image segmentation or object detection models, which the operator can then confirm or reject one by one. While this confirmation or rejection operation is simple, it constitutes a constraint on the expression of the candidate task and falls within the scope of the interactive cue constraints involved in this invention. The task publishing module stores the association between the generated candidate version, the manually confirmed version, and the final task version. As long as the final region passes the physical surface association, no-spray semantics, and robot serviceability checks, the system can directly generate and execute the spraying task after confirmation, without requiring additional boundary drawing by the operator.

[0127] Example 6: In this example, the human-machine interface displays the vehicle and the outline of identifiable components. The operator clicks on the left front door and marks a general location within it. The system uses the left front door component identifier as a range constraint and the marked location as a local positive prompt. The task generation module obtains the component candidate range from the vehicle model database, automatic component recognition results, or a universal door template, and then determines the damage area within that range based on image texture. Even if the database does not contain a precise vehicle model, universal component boundaries can be established using door seams, window frames, and handle features.

[0128] The component template is merely a source for region generation; it still needs to be ultimately registered with the current vehicle using images or geometry. If the registration residual exceeds the limit, the entire template cannot be directly executed; instead, it enters a re-identification or manual correction state. This embodiment eliminates the need for the operator to precisely draw complete boundaries on each component; instead, the painting area associated with the current physical vehicle can be formed using only component identifiers and rough location hints. This reduces interaction complexity, while registration residual verification ensures the reliability of task association.

[0129] Example 7: In this example, a fixed camera is used to acquire vehicle images within the current service range of the robotic arm. During installation and debugging, calibration points or calibration plates are set at multiple known locations to establish a homography mapping from the display position to the robotic arm's working plane. For the gently curved surface on the side of the vehicle, the service range is divided into multiple local regions and mappings are established for each region, with weighted interpolation used for continuous transition between regions. This method is a low-cost, single-view mapping path that does not require multiple observations for reverse indexing, resulting in lower hardware and debugging costs.

[0130] After the operator applies the coating, the algorithm generates the final mask in the image. The system does not need to convert each finger trajectory point into robot coordinates beforehand; instead, it can perform mapping only on the final mask boundaries, the intersections of the planned scan lines, or the trajectory sampling points. This reduces computational load while still establishing the task association between the final area and the physical vehicle. The spray gun's end-of-field ranging device maintains the spray distance to the vehicle surface in real time. If a trajectory point is located in the calibration extrapolation zone, the mapping residual is too large, or the ranging fails, the corresponding trajectory cannot be published. This embodiment can be implemented using a common industrial camera and a simple calibration process, without requiring multi-view reconstruction or expensive 3D scanning equipment, making it a preferred implementation for low-cost deployment scenarios.

[0131] Example 8: In this example, when the system already has multi-observation images, registration depth, and inverse indexes, these data can be directly called to inversely map the cue constraints or candidate regions to common reference coordinates, forming a 3D surface mask. This example is compatible with multi-observation 3D interactive localization technology, but the focus of this invention remains on the relationship between manual prompts and automatically generated final regions, rather than the method of acquiring multi-observation data itself.

[0132] When the system already has a computer-aided design model or digital twin surface of the current vehicle, a rendered image or camera image can be displayed on the screen, and the automatically segmented region can be attached to the model surface through image-model registration. Alternatively, the operator can directly click on the mesh unit or component on the model surface, and the system will automatically generate the final region in its neighborhood. The final region can be represented as a set of mesh patches, a subset of point clouds, a parametric surface interval, a 3D mask, or a surface attribute field. The downstream spraying control module can read the data through its own interface, without limiting the invention to a specific object file format. When the system has multiple observation data, it can also utilize the reverse index from the display position to the original observation data to perform region generation or suggestive constraint application on the highest resolution original image, thereby improving boundary accuracy, especially suitable for repair scenarios involving complex surfaces or minor damage.

[0133] Example 9: In this example, the operator first uses their finger to continuously apply paint to the core area to be repaired on the displayed vehicle screen, and then uses another color or another tool to draw the transition area between the old and new coatings. For example... Figure 7 As shown, the system generates positive prompts for the core area and prompts for the outer boundary of the transition area, automatically closes the break, smooths the boundary of the transition area, and deducts prohibited painting areas such as windows, lights, and trim strips.

[0134] The system generates layered spraying coverage according to the core area, transition area and spray swath footprint. After the smearing is completed and the input of the transition area is finished, if the data version, regional confidence, mapping accuracy, robot reachability and safety chain all pass, the system no longer requires the operator to perform other operations and directly controls the manipulator to perform spraying. This embodiment corresponds to a streamlined operation chain in actual products: smearing the area to be repaired, delineating the transition area, and automatic spraying by the robot. No matter pixel mask, automatic segmentation, boundary extraction, coordinate transformation, surface mapping or streaming trajectory is adopted internally, as long as the interaction prompt actually constrains the final task area and triggers automatic spraying, the technical solution of the present invention can be applied. In the automatic generation mode of the transition area, the system calculates the geodesic distance d(x) based on the boundary of the core area, sets the area where 0≤d(x)≤d1 as the color paint extension area, and sets the area where d1≤d(x)<d2 as the feathering or fusion area. d1 and d2 can vary with the curvature, spray swath direction, sheet metal edge and coating type, so that a transition area meeting the process requirements can still be generated when the operator only smears the core area.

[0135] To prove that the present invention can be implemented without significantly increasing the cost of the existing system, the following preferred minimum configuration is provided: an ordinary industrial color camera covering the service range of the manipulator; a touch display outside the explosion-proof area or an ordinary display with a mouse; an industrial personal computer running image processing and task planning software; a manipulator and a spray gun; and an optional end ranging module. The software process is: acquire and display the vehicle image; receive finger smearing; convert the smearing into foreground seeds; generate the final core area by region growing, graph cut or edge adsorption; generate the transition area by morphological dilation or surface distance; map the final area to the manipulator workspace through installation period calibration or calibration grid interpolation; generate parallel reciprocating trajectories; verify the reachability, prohibited areas and spray swath footprint; and execute after passing the verification.

[0136] Another preferred minimum configuration is: a tablet computer displaying a pre-stored 3D vehicle model or 2D vehicle model drawing; several sections of special color tape, reflective tape or other detectable external objects; an ordinary color camera; an industrial personal computer storing the vehicle model and the relationship between the markers and the vehicle; a manipulator and a spray gun. Maintenance personnel select an area on the surface of the pre-stored vehicle model, the ordinary camera identifies the tape or external objects to obtain the pose of the current vehicle or the area to be repaired, and the system migrates the model area to the vehicle surface before generating a trajectory and executing it. This configuration does not require on-site 3D scanning, and its cost is similar to that of the on-site image route circling scheme. None of the above solutions relies on large artificial intelligence models, expensive 3D scanners or multiple times of manual precise edge drawing, and the implementation cost is close to that of the direct screen circling scheme.

[0137] Embodiment 10: This embodiment further elaborates on other variations and equivalent implementations:

[0138] The screen referred to in this invention includes a touchscreen, a monitor, a mobile phone, a tablet computer, a projection interface, an augmented reality display device, or an equivalent interface capable of displaying target representation data and receiving directional input. Target representation data can be two-dimensional images or three-dimensional data obtained on-site, or it can be pre-stored vehicle images, three-dimensional vehicle models, two-dimensional unfolded diagrams, component schematics, parametric surfaces, mesh diagrams, or repair work order diagrams; it is not required that the target representation has been bound to the current vehicle during manual selection. Interactive input can be performed by a local operator or generated by a remote terminal and transmitted as pre-annotated images, strokes, masks, component identifiers, model coordinates, relative coordinates of markers, trajectory files, or interface messages.

[0139] The spraying method used in this invention includes liquid coatings, primers, paints, varnishes, protective coatings, adhesives, or other fluids or powders that can be applied to a target surface by a robot to form a covering. Without altering the core technological relationships, it can also be used for robotic operations such as grinding, cleaning, inspection, and polishing, where the task area is determined based on on-screen prompts.

[0140] The determination of candidate task representations in this invention includes both methods that are entirely automatically generated or modified by algorithms, and methods that directly employ manual selection of a range on the target representation data before performing model surface attachment, physical target association, coverage generation, trajectory generation, or online control. It also includes methods where algorithms or external markers first generate candidate results, with manual verification, elimination, local modification, or selection of one of multiple candidates performed only once. Whether manual strokes are saved or the final boundaries are re-divided does not affect the aforementioned technical relationships.

[0141] The external markings referred to in this invention include external physical markings as well as non-physical cues such as projected patterns and light spots. External physical markings are objects or materials temporarily placed or attached and recognizable by visual devices, including but not limited to specially colored tape, reflective tape or other reflective materials, coded labels, printed marks, stickers, magnetic attachments, calibration plates, positioning clamps, blocks, and external objects with specific colors, textures, shapes, depths, or spatial arrangements. External markings can be used for vehicle or component positioning, and also to indicate the core area to be repaired, boundaries, center lines, transition areas, areas where painting is prohibited, or their relative positions.

[0142] The task representation referred to in this invention includes explicit regions, as well as effective operating ranges implicitly determined by centerline and width, trajectory and spray width parameters, spray width footprint union, spray contribution range, or online control range, without requiring the generation, saving, or retrieval of independent region objects as a necessary condition.

[0143] The robot control described in this invention does not require all task data to be sent at once. It can be sent as a whole, or it can be sent while planning, streamed by trajectory segment, or the control target can be formed in real time through visual feedback; display and input, task generation, task issuance, and robot control do not need to be located on the same device.

[0144] In an implementation of an external task instruction data entry method independent of screen interaction, the damage detection device outputs the local coordinates and damage range of the vehicle component, or the maintenance work order terminal allows manual input of the center point, length, width, and direction, or the upstream system only sends the coverage unit, tool pose, or robot trajectory. The aforementioned data does not necessarily rely on manual screen selection. After obtaining the above data, the task entry parsing module first parses at least two of the following: data source, target or component identity, coordinate system or relative reference, region semantics, model or surface version, and task version, and merges them with the current vehicle or component positioning data. If the correspondence can be verified, this data is used as the basis for the candidate task expression; if it cannot be verified, at least one of the following is used to re-establish the association between the candidate task expression and the current physical target: the current observed image, natural features, external entity markers, vehicle or component reference, and fixture reference. For external coverage or trajectory input, the task generation module obtains the implicit effective working range from its centerline width, spray footprint, or effective range. The task association module attaches this range to the current physical surface, and the task publishing module verifies the target identity, region semantics, prohibited areas, surface version, and robot serviceability. In any case, external coordinates, overlays, or trajectory data that have not been parsed, associated, and published without verification shall not be sent directly to the robot controller, even if it contains robot coordinates, tool poses, or approval marks.

[0145] In one distributed implementation, the screen interaction terminal, external detection or task generation device, target association node, task publishing node, and robot control node are independent of each other; the message carries at least two of the following: task identifier, entry type, data source, target or component identity, coordinate system or relative reference, region semantics, model or surface version, task version, and publishing status. Nodes may not be connected to the network in real time, but backend nodes perform independent verification of the current physical target, version, and serviceability. In the above distributed implementation, upstream devices may only transmit model coordinates, texture coordinates, component local coordinates, region masks, overlay units, trajectories, or robot tasks derived from manual selection, without transmitting on-site images, original interaction strokes, or intermediate processing records; downstream devices establish technical associations based on the determination or constraint effect of the received data on the final working space range, without relying on the ability to reverse-engineer the original selection process.

[0146] This invention can also be implemented as a control device or robot system including a processor, memory, screen interaction interface, external task data interface, field positioning interface, robot task output interface, and robot control interface; it can also be implemented as a non-transitory computer-readable storage medium, computer program product, or distributed processing system. Program instructions cause the processor to perform entry point parsing, candidate task generation, physical target association, release verification, and execution control, and actually change the robot's trajectory, tool pose, spraying medium start / stop, and material deposition. While retaining the low-cost approach of screen interaction, the above processing prevents external coordinates, overlays, or trajectories from bypassing the current physical target association and task release verification by omitting screen steps. This helps reduce target mismatch, unauthorized entry into restricted areas, execution of incorrect versions, and accidental spraying, and allows the data processing results to have a direct technical effect on the physical robot's movement and the spraying process.

[0147] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating and executing spraying tasks based on interactive prompts, characterized in that, include: Obtain task entry data, which includes interactive prompt data generated based on target representation data or external task indication data; wherein, the external task indication data needs to be parsed to determine at least two of its data source, target or component identity, coordinate system or relative reference before it can enter subsequent processing; Candidate task expressions are determined based on the task entry data, and the candidate task expressions include explicit task areas or implicit effective work ranges; wherein, when the task entry data includes the interactive prompt data, the boundaries of the candidate task expressions are constrained by the interactive prompt data; The candidate task representation is associated with the physical surface of the current physical target to obtain the final task representation; The final task expression is validated and, upon successful validation, robot-executable task data is generated to control the robot to perform tasks on the current physical target.

2. The method according to claim 1, characterized in that, The interactive prompt data includes at least one of positive prompt data, negative prompt data, or stop boundary data; the interactive prompt data is formed by one or more, continuous or segmented, inputs from the same or different interactive terminals, and is combined before or during the determination of the candidate task expression; Based on the task entry data, candidate task representations are determined, including: Based on the target representation data and interactive prompt data, perform region generation processing to generate an explicit task region; Specifically, when the interactive prompt data includes positive prompt data, the explicit task area includes the physical location corresponding to the positive prompt data; when the interactive prompt data includes negative prompt data, the explicit task area excludes the physical location corresponding to the negative prompt data; when the interactive prompt data includes stop boundary data, the boundary of the explicit task area is limited by the stop boundary data.

3. The method according to claim 1, characterized in that, Based on the task entry data, candidate task representations are determined, including: Generate at least one of the following: spraying centerline, coverage unit, or spraying trajectory based on the task entry data; The union of the spray pattern footprints corresponding to the spray centerline, the covering unit, or the spray trajectory is determined as the implicit effective working range; The release verification includes: when the task entry data includes interactive prompt data, verifying whether the implicit valid job range meets the preset prompt consistency condition; and verifying whether the implicit valid job range meets the prohibited job condition.

4. The method according to claim 1, characterized in that, The target representation data includes pre-stored target representation data; Associating the candidate task representation with the physical surface of the current physical target includes: Identify external entity markers set on the current physical target or its associated environment, or identify the natural features of the current physical target; Based on the external entity markers or natural features, the candidate task representations in the pre-stored target representation data are mapped to the physical surface of the current physical target; The final task expression's job semantics and final job scope are determined based on at least one of the following: screen task selection instructions, external task semantics, verified preset rules, automatic confirmation results, or semi-automatic confirmation results.

5. The method according to claim 4, characterized in that, The external entity markings include at least one of the following: special color tape, reflective material, coded label, printed mark, sticker, magnetic attachment, calibration plate, positioning clamp or stop; The external entity markers are used to enclose, approximately enclose, or segment the repair area of ​​the current physical target, or to indicate the pose of the current physical target. The external entity markers are identified by a vision device, and the machine candidate boundaries or marker relative coordinates are extracted. The process of determining the job semantics and final job scope of the final task expression includes: The machine candidate boundary or its corresponding result on the pre-stored target representation data is displayed on the human-computer interaction interface; The screen task selection instruction received by the operator, which selects between task type, core area to be repaired, transition area, prohibited area, or candidate area, is used to form the final task expression.

6. The method according to claim 1, characterized in that, Perform release validation on the final task expression, including: Verify whether the correlation between the target representation data, candidate task expression, and physical surface belongs to a compatible target version; Verify whether the work scope corresponding to the final task expression intrudes into the prohibited work area; Verify the serviceability of the robot in expressing the final task; When the release verification passes, the robot automatically releases the task data and controls the robot to perform the operation without receiving additional manual confirmation instructions.

7. The method according to claim 2, characterized in that, The target representation data includes initial display data; Based on the target representation data and interactive prompt data, a region generation process is performed to generate an explicit task region, including: A rough work area is determined based on the initial display data and interactive prompt data; Determine the supplementary acquisition field of view or sensor pose based on the rough working area; Control the vision module or robotic arm to move to the supplementary acquisition field of view or sensor pose to acquire supplementary acquisition data; Based on the supplementary collected data, the region generation process is re-executed to obtain the explicit task region; The boundaries of the regenerated explicit task region must not exceed the permitted physical range corresponding to the rough task region.

8. The method according to claim 1, characterized in that, The robot's executable task data includes at least one of the following positional relationship expressions: A homogeneous transformation matrix containing Cartesian coordinate components and tool orientation; Surface parameter coordinates and distance along the surface normal; Mesh patch identifiers and in-plane coordinates; The local coordinates of the component or the relative coordinates of the marker; Robot joint target; Image spatial control increment based on real-time image feedback; The robot's executable task data is generated and sent in a streaming manner by region block, trajectory segment, or image feedback during the robot's operation.

9. A spraying task generation and execution system based on interactive prompts, characterized in that, include: The task entry acquisition module is used to acquire task entry data, which includes interactive prompt data generated based on target representation data or external task indication data. The task generation module is used to determine candidate task expressions based on the task entry data. The candidate task expressions include explicit task regions or implicit effective work ranges. When the task entry data includes interactive prompt data, the boundaries of the candidate task expressions are constrained by the interactive prompt data. The task association module is used to establish a task association between the candidate task expression and the physical surface of the current physical target to obtain the final task expression; The task publishing module is used to perform publishing verification on the final task expression; The execution control module is used to generate robot-executable task data after the verification is passed, so as to control the robot to perform operations on the current physical target.

10. A control device, characterized in that, include: One or more processors; At least one memory stores instructions that can be executed by the one or more processors; Robot task output interface; When the instruction is executed by the one or more processors, the control device performs the spraying task generation and execution method based on interactive prompts as described in any one of claims 1 to 8, and sends the robot-executable task data to the robot through the robot task output interface.

Images