Component plugging guide method, electronic device, and computer-readable storage medium
By using AR devices to guide component insertion and employing socket detection and pose tracking technologies, the insertion mode is adaptively selected and displayed for guidance, which solves the problems of low efficiency and accuracy in component insertion and achieves efficient and accurate insertion operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2026-01-26
- Publication Date
- 2026-06-19
AI Technical Summary
In the existing technology, the component insertion operation is inefficient and inaccurate, mainly because the connection between the socket and the wire needs to be determined by visual inspection or by checking the process documents, resulting in low insertion efficiency and low accuracy.
An AR device is used to guide the insertion of components. The device detects the socket and tracks its position in response to user operations. It selects the target insertion mode based on the number of sockets and quickly determines the information of the socket and wire to be inserted, including the hole-to-wire mode and the wire-to-hole mode, through display guidance processing. The insertion method is adaptively adjusted to improve efficiency and accuracy.
It improves the efficiency and accuracy of component insertion, especially when the hole is severely obstructed during the insertion process. Through adaptive insertion mode and display guidance, it ensures that the insertion process is efficient and accurate.
Smart Images
Figure CN122244144A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a component insertion and guidance method, an electronic device, and a computer-readable storage medium. Background Technology
[0002] In industry, mating refers to the process of accurately connecting and aligning interfaces (plugs and sockets) of multiple industrial devices or components using physical means. In the context of precision components, this operation primarily refers to the action of inserting a thin wire into a socket, specifically the process of inserting the wire into the corresponding numbered socket.
[0003] Currently, component insertion is mainly done manually. However, since the connection between the socket and the wire requires manual visual inspection or checking the process documents to determine the corresponding socket for the wire, the insertion efficiency and accuracy are low.
[0004] Therefore, a component insertion guidance method is needed to improve the accuracy and efficiency of component insertion. Summary of the Invention
[0005] This application provides a component insertion guidance method, an electronic device, and a computer-readable storage medium, which can improve the efficiency and accuracy of component insertion.
[0006] To solve the above-mentioned technical problems, one technical solution adopted in this application is: providing a component insertion guidance method, the component insertion guidance method comprising: responding to user operation on the AR device, performing component socket detection and pose tracking processing based on the acquired current scene image to obtain the currently detected socket of the component in the current scene image and the current pose information of the component in the camera coordinate system; selecting a target insertion mode from preset insertion modes according to the number of sockets of the currently detected socket; responding to the target insertion mode being a hole lead mode, according to the current... The system detects whether there is a hole sequence in the pre-detection socket and determines the target wire information of the pre-detection socket based on the acquired prior information, and performs display guidance processing on the pre-detection socket and the target wire information of the pre-detection socket. In response to the target connection mode being wire-lead mode, the system determines the prior socket information of the pre-detection wire from the prior information based on the acquired pre-detection wire, and determines the target socket of the pre-detection wire based on the current pose information, the prior socket information of the pre-detection wire, and the currently detected socket, and performs display guidance processing on the pre-detection wire and the target socket of the pre-detection wire.
[0007] In one embodiment, the step of determining the target wire information of the socket to be plugged in based on whether there is a hole sequence in the currently detected socket and the acquired prior information includes: in response to the existence of a hole sequence in the socket to be plugged in, determining the target wire information of the socket to be plugged in from the prior information based on the hole sequence of the socket to be plugged in; in response to the absence of a hole sequence in the socket to be plugged in, determining the model coordinates of the original socket of the component in the virtual simulation model from the prior information, and matching the pixel coordinates of the socket to be plugged in the pixel coordinate system with the model coordinates of each original socket in the virtual simulation model to obtain the target wire information of the socket to be plugged in.
[0008] In one embodiment, the step of matching the pixel coordinates of the hole to be inserted in the pixel coordinate system with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted includes: transforming the model coordinates of each original hole in the virtual simulation model according to the current pose information of the component in the camera coordinate system to obtain the three-dimensional coordinates of each original hole in the camera coordinate system; transforming the three-dimensional coordinates of each original hole in the camera coordinate system using the obtained camera intrinsic parameters to obtain the pixel coordinates of each original hole in the pixel coordinate system; selecting the original hole whose pixel distance from the pixel coordinates of the original hole to the pixel coordinates of the hole to be inserted is less than a pixel distance threshold; determining the target wire information of the selected original hole from the prior information, and using the target wire information of the selected original hole as the target wire information of the hole to be inserted.
[0009] In one embodiment, the prior socket information includes the prior socket of the wire to be connected and the model coordinates of the prior socket in the virtual simulation model. The step of determining the target socket of the wire to be connected based on the current pose information, the prior socket information of the wire to be connected, and the currently detected socket includes: transforming the model coordinates of the prior socket in the virtual simulation model based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system; and in response to the existence of a detected socket in the currently detected sockets whose pixel distance to the prior socket is less than a pixel distance threshold, determining the corresponding detected socket as the target socket of the wire to be connected.
[0010] In one embodiment, after the step of converting the model coordinates of the prior socket in the virtual simulation model based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system, the method further includes: in response to the absence of a detected socket in the currently detected sockets whose pixel distance to the prior socket is less than the pixel distance threshold, selecting a target sub-plugging mode from the sub-modes corresponding to the wire-lead mode based on the number of sockets of the currently detected sockets in the current scene image; in response to the target sub-plugging mode being a position guidance mode in the wire-lead mode, determining the target socket of the wire to be connected based on the position guidance mode; and in response to the target sub-plugging mode being a region guidance mode in the wire-lead mode, performing display guidance processing based on the region guidance mode.
[0011] In one embodiment, the step of determining the target socket of the wire to be connected according to the position guidance mode includes: determining the target pose information of the component in the camera coordinate system by using an improved random sampling consensus algorithm; transforming the model coordinates of the prior socket in the virtual simulation model using the target pose information and the camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system; and determining the corresponding detection socket as the target socket of the wire to be connected in response to the existence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than the pixel distance threshold.
[0012] In one embodiment, the step of performing display guidance processing according to the region guidance mode includes: determining a target inserted hole that matches the acquired inserted hole from each inserted hole in the current scene image; determining the minimum circumcircle of the inserted hole and the target inserted hole according to the pixel coordinates of the target inserted hole in the pixel coordinate system; and performing display guidance processing on the inserted hole, the target inserted hole, and the minimum circumcircle.
[0013] In one embodiment, the step of selecting a target plugging mode from a preset plugging mode based on the number of plugs in the currently detected plug includes: in response to the number of plugs in the currently detected plug being greater than or equal to a first plug threshold, determining the target plugging mode as the hole lead wire mode in the preset plugging modes; and in response to the number of plugs in the currently detected plug being less than the first plug threshold, determining the target plugging mode as the wire lead hole mode in the preset plugging modes.
[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above-mentioned component insertion guidance method.
[0015] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium including program data, which, when executed by a processor, is used to implement the above-mentioned component insertion and guidance method.
[0016] The above scheme, in response to user operation of the AR device, performs component socket detection and pose tracking processing based on the acquired current scene image to obtain the currently detected socket of the component in the current scene image and the current pose information of the component in the camera coordinate system; selects a target plugging mode from preset plugging modes based on the number of sockets of the currently detected socket; if the target plugging mode is a hole-lead mode, determines the target wire information of the hole to be plugged based on whether the hole sequence exists in the currently detected socket and the acquired prior information, and performs display guidance processing on the hole to be plugged and the target wire information of the hole to be plugged; if the target plugging mode is a wire-lead mode, determines the prior socket information of the wire to be plugged based on the acquired wire to be plugged from the prior information, determines the target socket of the wire to be plugged based on the current pose information, the prior socket information of the wire to be plugged, and the currently detected socket, and performs display guidance processing on the wire to be plugged and the target socket of the wire to be plugged. Considering that in actual insertion processes, as the number of holes for component insertion increases, there is a possibility that the inserted wires may block uninserted holes. If the same insertion method is used for the entire process, it will result in low efficiency. To address this, the embodiments of this application adaptively determine the target insertion mode by detecting the number of holes currently being detected, and perform insertion guidance processing according to the insertion method corresponding to the target insertion mode. This can improve both the efficiency and accuracy of component insertion. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:
[0018] Figure 1 This is a schematic flowchart of an exemplary embodiment of the component insertion guidance method shown in this application; Figure 2 yes Figure 1 A flowchart illustrating an exemplary embodiment of step S130 in the component insertion guidance method is shown. Figure 3 yes Figure 1 A flowchart illustrating an exemplary embodiment of step S140 in the component insertion guidance method is shown. Figure 4 This is a schematic diagram of an exemplary embodiment of the AR device shown in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 6 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0020] First, it's important to clarify that in industrial applications, mating refers to the process of accurately connecting and aligning interfaces (plugs and sockets) of multiple industrial devices or components using physical methods. In the context of precision components, this operation primarily refers to the act of inserting long, thin wires into the corresponding sockets; specifically, it's the process of inserting wires into the sockets with their corresponding serial numbers. Taking electrical connectors as an example, the entire mating process involves inserting several wires one by one into the sockets with their corresponding serial numbers. Generally, an electrical connector contains a dozen to several dozen sockets, and the wires to be mated consist of several wire bundles (combined together). Each bundle has a corresponding bundle number label, and the wires within each bundle are of different colors to distinguish them. The serial number corresponding to the wire to be mated in each socket is recorded in the process documentation. For example, "Socket 1 - Wire Bundle A2 (Red)" means that the red wire from wire bundle A2 should be inserted into socket 1.
[0021] Currently, component insertion is mainly done manually. However, since the connection between the socket and the wire requires manual visual inspection or checking the process documents to determine the corresponding socket for the wire, the insertion efficiency and accuracy are low.
[0022] Based on this, this application proposes a component insertion guidance method, details of which can be found in [reference needed]. Figure 1 , Figure 1 This is a schematic flowchart of an exemplary embodiment of the component insertion guidance method shown in this application.
[0023] The execution entity of the component insertion guidance method can be a terminal device, a server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The execution entity of the component insertion guidance method can also be an AR (Augmented Reality) device. In some possible implementations, this component insertion guidance method can be implemented by the processor calling computer-readable instructions stored in memory.
[0024] In this application embodiment, an AR device is used as the execution subject for description. Specifically, the component insertion guidance method of this application embodiment includes the following steps: S110, in response to the user's operation on the AR device, performs component socket detection and pose tracking processing based on the acquired current scene image, and obtains the current detected socket of the component in the current scene image and the current pose information of the component in the camera coordinate system.
[0025] User interaction with the AR device can involve clicking or touching the connection guide button on the device. The AR device receives the user's input and acquires an image of the current scene. This image can be directly captured by the AR device (i.e., an image taken when the AR device is directly facing the component's connector surface), or it can be acquired by the AR device from another camera. (The other camera captures the current scene image when it is directly facing the component's connector surface.)
[0026] As an example of socket detection methods, an AR device can perform feature extraction on the current scene image and compare the extracted features with preset features to determine the socket information of components in the current scene image. Socket information includes the socket itself and its sequence. As another example, an AR device can call a target socket detection model, inputting the current scene image into the target socket detection model to obtain the currently detected sockets predicted by the model.
[0027] The method for determining the target socket detection model involves acquiring a training image set and the annotation information of the component hole positions in each training image. The training images from the training image set, along with the annotation information of the component hole positions in each training image, are then input into an initial socket detection model for training, resulting in a target socket detection model that meets the requirements. The socket detection model can be a Yolov8 network model. The annotation information can be the socket category of the hole position region.
[0028] The method for obtaining the annotation information of the component hole positions in the training image set and each training image is as follows: The virtual simulation model of the component is imported into a virtual engine system, such as the Unity virtual engine, and a virtual camera is set up in the virtual engine system for imaging the virtual simulation model of the component. Multiple training images are obtained by fine-tuning the angle of the virtual camera facing the component's socket surface and the lighting conditions. Then, the training images are filtered and binarized to obtain processed training images. The processed training images can clearly distinguish between the socket and the mating surface. Connected component extraction and noise removal are performed on the processed training images to obtain the hole positions in the training images. The hole positions in the training images are then annotated to obtain the annotation information of the component hole positions in each training image.
[0029] For pose tracking, the current pose information of the component in the camera coordinate system can be determined through a target pose tracking model. Specifically, the target pose tracking model is acquired by importing a virtual simulation model of the component into a virtual engine system. Within the virtual engine system, a virtual camera is set up for imaging the virtual simulation model, and a depth camera is set up to acquire the depth map of the virtual simulation model. The virtual camera's viewpoint is adjusted to acquire first training images at different angles, and the depth image's viewpoint is adjusted to acquire second training images at different angles. These first and second training images are then input into the Foundation-pose network framework for training to obtain the target pose tracking model.
[0030] S120: Select the target plugging mode from the preset plugging modes based on the number of plugs currently detected.
[0031] The preset mating modes include wire-lead mode and hole-lead mode. Hole-lead mode involves locating the corresponding wire through the hole to be mated, primarily occurring in the early to mid-stages of the mating process. At this stage, there are fewer wires on the component, and the mating surface is less obstructed. Wire-lead mode involves locating the corresponding target hole on the component through the wire to be mated, primarily occurring in the later stages of the mating process. At this stage, a large number of wires are mated on the mating surface, resulting in significant obstruction.
[0032] The AR device determines whether the number of currently detected sockets is greater than the first socket threshold. If the number of currently detected sockets is greater than or equal to the first socket threshold, the target plugging mode is determined to be the hole lead wire mode in the preset plugging mode. If the number of currently detected sockets is less than the first socket threshold, the target plugging mode is determined to be the wire lead hole mode in the preset plugging mode.
[0033] S130, in response to the target insertion mode being hole lead mode, determines the target wire information of the hole to be inserted based on whether the hole sequence exists in the currently detected hole and the acquired prior information, and performs display guidance processing on the hole to be inserted and the target wire information of the hole to be inserted.
[0034] The hole sequence is the serial number of the socket. When the same socket is detected among the components, the socket can be further distinguished according to the hole sequence.
[0035] Prior information is used to quickly locate wires via sockets or vice versa. AR devices can use process documents, AR commands, or a combination of both as prior information. The process document includes the correspondence between socket numbers and wire information. For acquiring AR commands, the AR device can import the virtual simulation model of the component into the virtual engine system, where prior information is collected and AR commands for the connection operation are designed. Specifically, in response to user clicks on the sockets of the component in the virtual engine system, the device obtains the model coordinates of each socket in the virtual simulation model—that is, the three-dimensional coordinates of each socket relative to the center of the virtual simulation model. These model coordinates are then bound to the corresponding wire information to obtain AR commands. These AR commands are stored for retrieval during real-time operations, allowing AR commands to be obtained directly from the virtual engine system, avoiding the tedious process of manual recording. The AR command includes each socket in the component, the model coordinates of the socket in the virtual simulation model, the socket sequence, and the wire information. The wire information includes the wire harness, the harness number, and the corresponding wire color. It should be noted that, to improve the efficiency of the connection operation, this embodiment can download the prior information corresponding to the component in advance when the component is detected. For example, it can download the AR command of the component in advance and parse the AR command to obtain each socket in the component, the socket sequence, the model coordinates of the socket, and the corresponding wire information.
[0036] The target wire information refers to the wire information that matches the socket to be plugged in. For example, if the socket to be plugged in is hole 1 and the wire information is wire harness A2 (red), it means that the red wire in wire harness A2 needs to be plugged into hole 1.
[0037] Display guidance processing refers to associating and displaying the socket to be plugged in and the target wire information of the socket on the AR device, so as to facilitate the staff to quickly find the corresponding wire information of the socket and perform the plugging operation based on the socket to be plugged in and the target wire information of the socket to be plugged in and displayed on the AR device.
[0038] When the AR device determines that the target insertion mode is the hole lead mode, it determines the target wire information of the hole to be inserted based on whether the hole sequence exists in the currently detected hole and the acquired prior information, and then displays and guides the hole to be inserted and the target wire information of the hole to be inserted.
[0039] S140, in response to the target insertion mode being wire-lead mode, the prior insertion information of the wire to be connected is determined from the prior information based on the acquired wire to be connected, the target insertion of the wire to be connected is determined based on the current pose information, the prior insertion information of the wire to be connected, and the currently detected insertion, and the display guidance process is performed on the wire to be connected and the target insertion of the wire to be connected.
[0040] The prior socket information includes the socket to which the wire to be connected needs to be plugged, the socket sequence, and the model coordinates of the socket in the virtual simulation model.
[0041] The AR device, upon determining the target connection mode to be a wire-lead mode, instructs the worker to place the wire to be connected under the AR device. The worker follows the instructions and places the wire under the AR device. The AR device, in response to acquiring an image including the wire to be connected, performs wire recognition processing on the image to obtain the wire to be connected. Based on the wire to be connected, it determines the prior socket information of the wire from prior information. Based on the current pose information, the prior socket information of the wire to be connected, and the currently detected socket, it determines the target socket of the wire to be connected and performs display guidance processing on the wire to be connected and its target socket. For the method of obtaining the wire to be connected through image wire recognition processing, the AR device can use OCR (Optical Character Recognition) technology to identify the wire harness, harness number, and corresponding wire color in the image.
[0042] As can be seen, the component insertion guidance method of this application responds to the user's operation on the AR device, performs component insertion hole detection and pose tracking processing based on the acquired current scene image, and obtains the current detected insertion hole of the component in the current scene image and the current pose information of the component in the camera coordinate system; selects a target insertion mode from the preset insertion mode according to the number of insertion holes of the current detected insertion hole; responds to the target insertion mode being the hole lead wire mode, determines the target wire information of the insertion hole based on whether the hole to be inserted exists in the current detected insertion hole and the acquired prior information, and performs display guidance processing on the insertion hole and the target wire information of the insertion hole; responds to the target insertion mode being the wire lead hole mode, determines the prior insertion hole information of the wire to be inserted from the prior information based on the acquired wire to be connected, determines the target insertion hole of the wire to be connected based on the current pose information, the prior insertion hole information of the wire to be connected, and the current detected insertion hole, and performs display guidance processing on the wire to be connected and the target insertion hole of the wire to be connected. Considering that in actual insertion processes, as the number of holes for component insertion increases, there is a possibility that the inserted wires may block uninserted holes. If the same insertion method is used for the entire process, it will result in low efficiency. To address this, the embodiments of this application adaptively determine the target insertion mode by detecting the number of holes currently being detected, and perform insertion guidance processing according to the insertion method corresponding to the target insertion mode. This can improve both the efficiency and accuracy of component insertion.
[0043] Based on the above embodiments, the component insertion guidance method of this application describes how to specifically determine the target wire information of the hole to be inserted in the lead wire mode. For details, please refer to [reference needed]. Figure 2 , Figure 2 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of step S130 in the component insertion guidance method. Specifically, step S130, which determines the target wire information of the hole to be inserted based on whether there is a hole sequence in the currently detected jacks and the acquired prior information, further includes: In step S210, in response to the existence of a hole sequence for the hole to be inserted, the target wire information of the hole to be inserted is determined from the prior information based on the hole sequence of the hole to be inserted.
[0044] In response to the AR device detecting the hole sequence of the socket to be inserted, it indicates that the AR device can directly determine the target wire information of the socket based on the hole sequence. Specifically, the AR device determines the target wire information of the socket to be inserted from prior information based on the hole sequence. For example, if the target wire information of the socket to be inserted is determined to be wire harness A3 (yellow) based on the hole sequence 2, it means that the yellow wire in wire harness A3 needs to be inserted into socket 2. Regarding the method of determining the hole sequence, the AR device can directly extract features from the socket to be inserted and determine the hole sequence by comparing the extracted features with preset hole sequence features. Alternatively, the AR device can input an image including the socket to be inserted into a trained hole sequence estimation model to obtain the hole sequence predicted by the hole sequence estimation model.
[0045] In step S220, in response to the absence of a hole sequence for the hole to be inserted, the model coordinates of the original hole of the component in the virtual simulation model are determined from the prior information, and the pixel coordinates of the hole to be inserted in the pixel coordinate system are matched with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted.
[0046] If the AR device fails to detect the hole sequence of the hole to be inserted, it indicates that the AR device cannot directly determine the target wire information of the hole from the prior information based on the hole sequence. To address this, the AR device can determine the original hole in the prior information that matches the hole to be inserted by matching the hole to be inserted with the original holes of the component in the prior information. Then, the wire information corresponding to the original hole in the prior information is used as the target wire information of the hole to be inserted. Specifically, the AR device determines the model coordinates of the original holes of the component in the virtual simulation model from the prior information, and matches the pixel coordinates of the hole to be inserted in the pixel coordinate system with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted.
[0047] The method for matching the pixel coordinates of the hole to be inserted in the pixel coordinate system with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted can be as follows: The model coordinates of each original hole in the virtual simulation model can be transformed based on the current pose information of the component in the camera coordinate system to obtain the three-dimensional coordinates of each original hole in the camera coordinate system; the three-dimensional coordinates of each original hole in the camera coordinate system can be transformed using the obtained camera intrinsic parameters to obtain the pixel coordinates of each original hole in the pixel coordinate system; the original hole whose pixel distance to the pixel coordinates of the hole to be inserted is less than a pixel distance threshold is selected from the pixel coordinates of each original hole; the target wire information of the selected original hole is determined from the prior information, and the target wire information of the selected original hole is used as the target wire information of the hole to be inserted.
[0048] Specifically, the three-dimensional coordinates of each original jack in the camera coordinate system can be represented as follows:
[0049] in, R represents the three-dimensional coordinates of each original socket in the camera coordinate system, and R,t represents the current pose information of the component in the camera coordinate system. This represents the model coordinates of each original socket in the virtual simulation model.
[0050] As can be seen, the component insertion guidance method of this application, in response to the existence of a hole sequence for the hole to be inserted, determines the target wire information of the hole to be inserted from prior information based on the hole sequence; in response to the absence of a hole sequence for the hole to be inserted, it determines the model coordinates of the original hole of the component in the virtual simulation model from the prior information, and matches the pixel coordinates of the hole to be inserted in the pixel coordinate system with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted. This method can quickly determine the target wire information of the hole to be inserted, thereby improving insertion efficiency and accuracy.
[0051] Based on the above embodiments, the component insertion guidance method of this application describes how to determine the target wire information of the hole to be inserted in the wire lead hole mode. For details, please refer to... Figure 3 , Figure 3 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of step S140 in the component insertion guidance method. Specifically, step S140, which determines the target socket for the wire to be connected based on the current pose information, the prior socket information of the wire to be connected, and the currently detected socket, further includes: Step S310: Based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters, the model coordinates of the prior socket in the virtual simulation model are transformed to obtain the pixel coordinates of the prior socket in the pixel coordinate system.
[0052] In wire-lead mode, even if the prior socket information of the wire to be connected is determined, there may be no matching socket among the currently detected sockets due to socket occlusion, leading to matching failure if a direct match is performed. Therefore, the AR device needs to first determine whether there is a matching socket among the currently detected sockets. For example, the AR device transforms the model coordinates of the prior socket in the virtual simulation model based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters, obtaining the pixel coordinates of the prior socket in the pixel coordinate system. Specifically, the AR device transforms the model coordinates of the prior socket in the virtual simulation model based on the current pose information of the component in the camera coordinate system to obtain the three-dimensional coordinates of the prior socket in the camera coordinate system; and transforms the three-dimensional coordinates of the prior socket in the camera coordinate system based on the camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system; then it determines whether there is a detection socket in the current detection socket whose pixel distance to the prior socket is less than the pixel distance threshold. If there is, it indicates that the detection socket of the wire to be connected is not blocked, and step S320 is executed; if not, it indicates that the detection socket of the wire to be connected is blocked, and step S330 is executed.
[0053] Step S320: In response to the presence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than the pixel distance threshold, the corresponding detection socket is determined as the target socket for the wire to be connected.
[0054] Step S330: In response to the absence of a detection socket in the current detection socket that has a pixel distance less than the pixel distance threshold from the prior socket, a target sub-plugging mode is selected from the sub-modes corresponding to the wire lead hole mode based on the number of current detection sockets in the current scene image.
[0055] The target sub-plugging modes include location-guided mode and region-guided mode. Location-guided mode is based on precise location-based guidance, while region-guided mode is based on region-based heuristic guidance.
[0056] Because hole obstruction exists in the wire-lead mode, to improve the accuracy and efficiency of the insertion, this embodiment of the application can determine which target sub-insertion mode to use in the wire-lead mode based on the number of holes currently detected. For example, the AR device determines whether the number of holes currently detected is less than a first hole threshold and greater than or equal to a second hole threshold, where the first hole threshold is greater than the second hole threshold. If the number of holes currently detected is less than the first hole threshold and greater than or equal to the second hole threshold, then step S340 is executed; if the number of holes currently detected is less than the second hole threshold, then step S350 is executed.
[0057] Step S340: In response to the position guidance mode in the target sub-plug mode being wire lead hole mode, determine the target plug hole for the wire to be connected according to the position guidance mode.
[0058] In response to the target sub-plugging mode being the position guidance mode in the wire-lead mode, the AR device determines the target pose information of the component in the camera coordinate system using the Modified Random Sample Consensus Algorithm (Modified RANSAC). The model coordinates of the prior plug in the virtual simulation model are transformed using the target pose information and camera intrinsic parameters to obtain the pixel coordinates of the prior plug in the pixel coordinate system. If a detected plug exists whose pixel distance to the prior plug is less than a pixel distance threshold, the corresponding detected plug is identified as the target plug for the wire to be connected. It should be noted that, in response to the absence of a detected plug whose pixel distance to the prior plug is less than a pixel distance threshold, this embodiment can also use a region-guided display guidance method.
[0059] Among them, the method of determining the target pose information of components in the camera coordinate system through the improved random sampling consensus algorithm Modified RANSAC has limitations due to the original sockets of the components. There are interfering matching terms; therefore, a ransac distance threshold can be set. The maximum number of iterations N is set to twice the minimum spacing between the sockets. This value can be obtained from the virtual engine system. Considering that the number of sockets on a component generally does not exceed 30, the maximum number of iterations N is set to 1000. The AR device starts from the original sockets of the component. The random selection of two original sockets without replacement can be represented as follows: , From the current detection socket Two detection ports are randomly selected from the pool, which can be represented as follows: , The Singular Value Decomposition (SVD) method was used to... and Component pose estimation is performed to obtain the estimated pose, i.e., the estimated R,t; the current detection socket is transformed based on the estimated pose, and the original socket is then processed. The distance to each currently detected jack is determined to be less than the ransac distance threshold. The original socket is identified, and the corresponding original socket and the currently detected socket are used as in-model point pairs; the above steps are iterated, and the estimated pose with the most in-model point pairs is determined as the optimal pose; the currently detected socket is transformed using the optimal pose, and the original socket is then used as the reference point. The distance to each currently detected jack is determined to be less than the ransac distance threshold. The original socket is identified, and the corresponding original socket and the currently detected socket are used as matching point pairs. The matching point pairs are then processed using the SVD method to obtain the target pose information.
[0060] The AR device transforms the model coordinates of the prior socket in the virtual simulation model using target pose information and camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system. Specifically, it transforms the model coordinates of the prior socket in the virtual simulation model using the target pose information of the component in the camera coordinate system to obtain the three-dimensional coordinates of the prior socket in the camera coordinate system. Then, it transforms the three-dimensional coordinates of the prior socket in the camera coordinate system using camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system.
[0061] Step S350: In response to the target sub-plug mode being the wire-lead hole mode, the display guidance process is performed according to the region guidance mode.
[0062] Considering that in the later stages of the insertion operation, due to the obstruction of a large number of already inserted wires, most sockets are no longer identifiable, this application embodiment uses heuristic region guidance based on the area where the socket to be inserted is located and the relative positional relationship between the socket to be inserted and other already inserted sockets. For example, in response to the region guidance mode in the target sub-insertion mode being the wire-lead socket mode, the AR device determines the target already inserted socket that matches the acquired socket to be inserted from among the already inserted sockets in the current scene image; determines the minimum circumcircle of the socket to be inserted and the target already inserted socket based on the pixel coordinates of the target already inserted socket in the pixel coordinate system; and performs display guidance processing on the socket to be inserted, the target already inserted socket, and the minimum circumcircle.
[0063] In the method of determining the target inserted hole that matches the acquired inserted hole from the existing inserted holes in the current scene image, the AR device obtains the model coordinates of the inserted hole and the model coordinates of each existing inserted hole from prior information, calculates the hole spacing between the model coordinates of the inserted hole and the model coordinates of each existing inserted hole, and determines the inserted holes with hole spacing less than or equal to a threshold as the target inserted hole that matches the inserted hole. For example, if a component has 10 inserted holes, inserted holes 1-10, and the existing inserted holes... Let (Socket 2, Socket 3, Socket 5, Socket 6) be the model coordinates of Socket 2 (4,5,6), Socket 3 (7,8,9), Socket 5 (10,11,12), and Socket 6 (13,14,15). The socket to be inserted is Socket 7, with model coordinates of (13,14,15). The AR device calculates the distances between Socket 7 and Socket 2, Socket 3, Socket 5, and Socket 6 respectively. If only the distance between Socket 5 and Socket 6 and Socket 7 is less than or equal to the threshold, then Socket 5 and Socket 6 are determined to be the target inserted sockets.
[0064] Specifically, the target socket satisfies the following formula:
[0065] in, This indicates that the target has been inserted. Indicates the set of sockets already inserted. This indicates a group of already inserted sockets. Indicates the socket to be inserted.
[0066] For determining the minimum circumcircle of the target hole and the existing hole based on their pixel coordinates in the pixel coordinate system, the AR device can determine the center and radius of the circle based on the pixel coordinates of the existing hole, and then determine the minimum circumcircle based on the center and radius. For example, if the pixel coordinates of hole 5 are (100, 200) and the pixel coordinates of hole 6 are (100, 200), then the center of the circle is ((100+120) / 2, (200+220) / 2) = (110, 210), and the radius is... ≈ 14.14mm, therefore, the minimum circumcircle is determined on the AR device with a center (110, 210) and a radius of 14.14. Regarding the determination of the pixel coordinates of the target inserted hole in the pixel coordinate system, the AR device transforms the model coordinates of the target inserted hole to pixel coordinates using the current pose information of the component and camera intrinsic parameters. Specifically, the AR device transforms the model coordinates of the target inserted hole based on the current pose information to obtain the 3D coordinates of the target inserted hole in the camera coordinate system; and transforms the 3D coordinates of the target inserted hole in the camera coordinate system based on the camera intrinsic parameters to obtain the pixel coordinates of the target inserted hole in the pixel coordinate system.
[0067] The AR device displays the hole to be inserted, the target already inserted hole, and the minimum circumscribed circle in a way that provides guidance. This allows staff to determine the area of the hole to be inserted on the AR device based on the minimum circumscribed circle, and to quickly perform the insertion operation based on the positional relationship between the hole to be inserted and the adjacent target already inserted hole.
[0068] As can be seen, the component insertion guidance method of this application transforms the model coordinates of the prior socket in the virtual simulation model based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters, to obtain the pixel coordinates of the prior socket in the pixel coordinate system; in response to the presence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than a pixel distance threshold, the corresponding detection socket is determined as the target socket of the wire to be connected; in response to the absence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than a pixel distance threshold, a target sub-insertion mode is selected from the sub-model corresponding to the wire lead model based on the number of current detection sockets in the current scene image; in response to the target sub-insertion mode being the position guidance mode in the wire lead mode, the target socket of the wire to be connected is determined according to the position guidance mode; in response to the target sub-insertion mode being the area guidance mode in the wire lead mode, display guidance processing is performed according to the area guidance mode. This application embodiment addresses the issue of severe obstruction during the later stages of insertion. It determines whether to use a position guidance mode or a region guidance mode based on the current number of detected sockets, and then performs display guidance processing based on the position guidance mode or the region guidance mode, which can improve insertion efficiency and accuracy.
[0069] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an exemplary embodiment of the AR device shown in this application. The AR device 400 includes a detection and tracking module 410, a plug-in mode selection module 420, a lead wire module 430, and a lead wire module 440. Specifically: The detection and tracking module 410 is used to respond to the user's operation on the AR device, and to perform component socket detection and pose tracking processing based on the acquired current scene image to obtain the current detected socket of the component in the current scene image and the current pose information of the component in the camera coordinate system.
[0070] The plug-in mode selection module 420 is used to select a target plug-in mode from preset plug-in modes based on the number of plugs currently detected.
[0071] The hole lead module 430 is used to respond to the target insertion mode being the hole lead mode, determine the target wire information of the hole to be inserted based on the presence of the hole sequence in the currently detected hole and the acquired prior information, and display the hole to be inserted and the target wire information of the hole to be inserted.
[0072] The wire-lead module 440 is used to respond to the target insertion mode being wire-lead mode, determine the prior insertion information of the wire to be connected from the prior information based on the acquired wire to be connected, determine the target insertion of the wire to be connected based on the current pose information, the prior insertion information of the wire to be connected and the currently detected insertion, and perform display processing on the wire to be connected and the target insertion of the wire to be connected.
[0073] In the above scheme, the AR device responds to user operations by performing component socket detection and pose tracking based on the acquired current scene image to obtain the currently detected socket and the current pose information of the component in the camera coordinate system. Based on the number of currently detected sockets, a target insertion mode is selected from preset insertion modes. If the target insertion mode is a hole-lead mode, the target wire information for the hole to be inserted is determined based on the existence of a hole sequence in the currently detected sockets and the acquired prior information. Display guidance processing is then performed on the hole to be inserted and its target wire information. If the target insertion mode is a wire-lead mode, the prior socket information for the wire to be connected is determined from the prior information based on the acquired wire to be connected. The target socket for the wire to be connected is determined based on the current pose information, the prior socket information of the wire to be connected, and the currently detected socket. Display guidance processing is then performed on the wire to be connected and its target socket. Considering that in actual insertion processes, as the number of holes for component insertion increases, there is a possibility that the inserted wires may block uninserted holes. If the same insertion method is used for the entire process, it will result in low efficiency. To address this, the embodiments of this application adaptively determine the target insertion mode by detecting the number of holes currently being detected, and perform insertion guidance processing according to the insertion method corresponding to the target insertion mode. This can improve both the efficiency and accuracy of component insertion.
[0074] The functions of each module can be found in the component insertion guidance method embodiment, and will not be repeated here.
[0075] To implement the component insertion guidance method of the above embodiments, this application proposes another electronic device, please refer to [link / reference needed]. Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application.
[0076] Electronic device 500 includes memory 501 and processor 502, wherein memory 501 and processor 502 are coupled together.
[0077] The memory 501 is used to store program data, and the processor 502 is used to execute the program data to implement the component insertion and guidance method of the above embodiment.
[0078] In this embodiment, processor 502 can also be referred to as CPU (Central Processing Unit). Processor 502 may be an integrated circuit chip with signal processing capabilities. Processor 502 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 502 can be any conventional processor.
[0079] This application also provides a computer-readable storage medium, such as Figure 6 As shown, the computer-readable storage medium 600 is used to store program data 601, which, when executed by a processor, is used to implement the component insertion guidance method as described in the method embodiments of this application.
[0080] The methods involved in the component insertion and guidance method embodiments of this application, when implemented as software functional units and sold or used as independent products, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0081] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A component insertion guidance method, characterized in that, The method is applied to an AR device, and the method includes: In response to the user's operation on the AR device, the component socket detection and pose tracking processing are performed based on the acquired current scene image to obtain the current detected socket of the component in the current scene image and the current pose information of the component in the camera coordinate system. Select the target plugging mode from the preset plugging modes based on the number of plugs currently detected; In response to the target insertion mode being the hole lead mode, the target wire information of the hole to be inserted is determined based on whether the hole sequence exists in the currently detected hole and the acquired prior information, and the hole to be inserted and the target wire information of the hole to be inserted are displayed and guided. In response to the target insertion mode being a wire-lead mode, the prior insertion information of the wire to be connected is determined from the prior information based on the acquired wire to be connected. The target insertion of the wire to be connected is determined based on the current pose information, the prior insertion information of the wire to be connected, and the currently detected insertion, and the display guidance process is performed on the wire to be connected and the target insertion of the wire to be connected.
2. The component insertion guidance method according to claim 1, characterized in that, The step of determining the target wire information of the socket to be inserted based on whether the socket to be inserted exists in the current detection socket and the acquired prior information includes: In response to the existence of a hole sequence in the hole to be inserted, the target wire information of the hole to be inserted is determined from the prior information based on the hole sequence of the hole to be inserted; In response to the absence of a hole sequence for the hole to be inserted, the model coordinates of the original hole of the component in the virtual simulation model are determined from the prior information, and the pixel coordinates of the hole to be inserted in the pixel coordinate system are matched with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted.
3. The component insertion guidance method according to claim 2, characterized in that, The step of matching the pixel coordinates of the hole to be inserted in the pixel coordinate system with the model coordinates of each original hole in the virtual simulation model to obtain the target wire information of the hole to be inserted includes: Based on the current pose information of the components in the camera coordinate system, the model coordinates of each original socket in the virtual simulation model are transformed to obtain the three-dimensional coordinates of each original socket in the camera coordinate system. The three-dimensional coordinates of each original jack in the camera coordinate system are transformed by the obtained camera intrinsic parameters to obtain the pixel coordinates of each original jack in the pixel coordinate system. Select the original sockets whose pixel distance from the pixel coordinates of the socket to be inserted is less than a pixel distance threshold. The target wire information of the selected original socket is determined from the prior information, and the target wire information of the selected original socket is used as the target wire information of the socket to be plugged in.
4. The component insertion guidance method according to claim 1, characterized in that, The prior socket information includes the prior socket of the wire to be connected and the model coordinates of the prior socket in the virtual simulation model. The step of determining the target socket of the wire to be connected based on the current pose information, the prior socket information of the wire to be connected, and the currently detected socket includes: Based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters, the model coordinates of the prior socket in the virtual simulation model are transformed to obtain the pixel coordinates of the prior socket in the pixel coordinate system. In response to the presence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than a pixel distance threshold, the corresponding detection socket is determined as the target socket for the wire to be connected.
5. The component insertion guidance method according to claim 4, characterized in that, After the step of transforming the model coordinates of the prior socket in the virtual simulation model based on the current pose information of the component in the camera coordinate system and the camera intrinsic parameters to obtain the pixel coordinates of the prior socket in the pixel coordinate system, the method further includes: In response to the absence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than the pixel distance threshold, a target sub-plugging mode is selected from the sub-modes corresponding to the wire lead hole mode based on the number of current detection sockets in the current scene image. In response to the target sub-plugging mode being the position guidance mode in the wire lead hole mode, the target plug hole of the wire to be connected is determined according to the position guidance mode; In response to the target sub-plugging mode being the area guidance mode in the wire hole mode, display guidance processing is performed according to the area guidance mode.
6. The component insertion guidance method according to claim 5, characterized in that, The step of determining the target socket for the wire to be connected based on the position guidance mode includes: The target pose information of the component in the camera coordinate system is determined by an improved random sampling consensus algorithm. The model coordinates of the prior emplacement in the virtual simulation model are transformed using the target pose information and the camera intrinsic parameters to obtain the pixel coordinates of the prior emplacement in the pixel coordinate system. In response to the presence of a detection socket in the current detection socket whose pixel distance to the prior socket is less than the pixel distance threshold, the corresponding detection socket is determined as the target socket for the wire to be connected.
7. The component insertion guidance method according to claim 5, characterized in that, The step of performing display guidance processing according to the region guidance mode includes: Determine the target inserted hole that matches the acquired inserted hole from among the inserted holes in the current scene image; The minimum circumcircle of the hole to be inserted and the target hole is determined based on the pixel coordinates of the target hole in the pixel coordinate system. The hole to be inserted, the target inserted hole, and the minimum circumscribed circle are displayed and guided.
8. The component insertion guidance method according to claim 1, characterized in that, The step of selecting a target plugging mode from preset plugging modes based on the number of currently detected plugs includes: In response to the current number of detected sockets being greater than or equal to a first socket threshold, the target plugging mode is determined to be the hole lead wire mode in the preset plugging modes; If the number of currently detected jacks is less than the first jack threshold, then the target plugging mode is determined to be the wire lead jack mode in the preset plugging modes.
9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the component insertion boot method as described in any one of claims 1-8.
10. A computer storage medium, characterized in that, include: The system stores program data, which, when executed by a processor, is used to implement the component insertion guidance method as described in any one of claims 1-8.