Image processing method and device and mark object with three-dimensional structure

Abstract

The invention provides an image processing method and device and a mark object with a three-dimensional structure. The method comprises the steps that an image processing device obtains a first image, when it is determined that the first image comprises a first target, key point detection is conducted on the first image, multiple first key points of the first target are determined, and the multiple first key points are used for determining internal and external parameter information of a first sensor or pose information of a to-be-shot object. As the first target is the mark object with the three-dimensional structure, the contour feature of the first target is unique and is not influenced by the change of the surface color and the texture feature, so that the first target has relatively strong anti-interference capability; therefore, even if the first image is formed when the first target is covered by coal dust or is influenced by illumination or is damaged, the first target included in the first image can be accurately identified, so that the calibration accuracy of the sensor can be effectively ensured; and the problem of picture splicing abnormity in the prior art can be solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and a marking object with a three-dimensional structure. Background Art

[0002] With the rapid development of video stitching technology, video stitching technology is applied in more and more fields (such as coal mining machine positioning in coal mine scenes or autonomous driving vehicle positioning in port and terminal scenes, etc.). For example, taking the coal mine scene as an example, the video stitching technology used in the coal mine scene can restore the continuity and spatial sense between multiple monitoring images in the coal mine scene, which helps to achieve the purpose of rapid and comprehensive observation of coal mining machines and coal wall conditions, greatly improving the efficiency of remote control coal mining.

[0003] Among them, video stitching technology includes two major functional modules: sensor calibration (such as camera calibration) and image stitching. For example, taking camera calibration as an example, camera calibration is the basis of video stitching technology. All correction, projection, stitching and fusion operations are based on accurate calibration. Therefore, the stability and accuracy of camera calibration are the decisive factors for the quality of image stitching.

[0004] At present, for coal mine scenes, one camera calibration scheme is: first install multiple 2D two-dimensional code identification cards (or can be called 2D two-dimensional code logo cards) on the coal mining machine. Afterwards, when images taken by multiple cameras are obtained, the computing device detects the two-dimensional code logo included in the image by using image detection technology and calculates the relative external parameters of multiple cameras online to complete the camera calibration. However, during actual comprehensive mining operations, coal dust is extremely serious, which will cause objects to be covered by oily dust in a short time and cannot be identified. For example, the 2D two-dimensional code identification card is blocked or damaged in a short time and becomes invalid, resulting in abnormalities in the picture splicing due to camera calibration errors (for example, video splicing will have ghosting, misalignment, incomplete subject, etc.). Summary of the invention

[0005] The embodiments of the present application provide an image processing method, an apparatus, and a marking object with a three-dimensional structure, which are used to ensure the effectiveness and accuracy of marking object recognition, thereby ensuring the accuracy of sensor calibration, and further effectively solving the problem of abnormal image splicing due to camera calibration errors in the prior art.

[0006] In the first aspect, the embodiment of the present application provides an image processing method, which can be performed by an image processing device or a component (such as a chip system or circuit, etc.) that can support the image processing device to implement the functions required by the method. Exemplarily, the image processing device can be a computing device having the functions required to implement the method, or it can also be a functional element (such as a plug-in, component or chip, etc.) having the functions required to implement the method, or it can also be any other device (such as a robot device, a cloud server or a vehicle-mounted device, etc.) having the functions required to implement the method, or it can also be a functional element set in a computing device or any other device. Optionally, taking the image processing device executing the image processing method as an example, in the method, the image processing device can first obtain a first image, wherein the first image is obtained by photographing the object to be photographed by the first sensor, and then, when it is determined that the first image contains the first target, the image processing device can perform key point detection on the first image to determine multiple first key points of the first target, wherein the first target is a landmark object with a three-dimensional structure, the first target is located on the object to be photographed, the first key point is used to characterize the structural features of the first target, and the multiple first key points are used to determine the internal and external parameter information of the first sensor, or to determine the posture information of the object to be photographed.

[0007] In the present method, the first target is a marking object with a three-dimensional structure, so the contour feature of the first target is unique and is not affected by the change of surface color and texture features. At the same time, the perspective projection or shadow formed by the surface shape or structural depth difference of the first target on the plane image can improve the recognition ability of the first target in a dirty situation, so that the first target has a relatively strong anti-interference ability and the first target has a relatively high robustness. Based on these characteristics of the first target, even if the first image acquired by the image processing device is formed when the first target is covered by dust (or dust or coal dust) or is exposed to light or damaged, the image processing device can quickly and accurately identify the first target included in the first image. Therefore, the method can ensure the effectiveness and accuracy of the recognition of the first target (i.e., the marking object with a three-dimensional structure), so that after the first target is identified, the internal and external parameter information of the sensor can be accurately calculated according to the multiple first key points of the first target, so that the accuracy of the sensor calibration can be effectively ensured, and then the problem of abnormal picture splicing due to camera calibration errors in the prior art can be effectively solved.

[0008] In one possible design, an image processing device performs key point detection on a first image to determine multiple first key points of a first target, including: the image processing device first performs key point detection on the first image to determine multiple first candidate key points of the first target, and can verify the multiple first candidate key points; when there are abnormal key points among the multiple first candidate key points, the image processing device eliminates the abnormal key points and can use the remaining multiple first candidate key points as the multiple first key points; or, when the number of the multiple first candidate key points is less than or equal to a quantity threshold, the image processing device supplements at least one missing key point among the multiple first candidate key points to obtain multiple second candidate key points, and can use the multiple second candidate key points as the multiple first key points.

[0009] In one example, when the number of the remaining multiple first candidate key points is less than or equal to a quantity threshold, the image processing device may also supplement at least one missing key point among the multiple first candidate key points to obtain multiple candidate key points, and may use the multiple candidate key points as multiple first key points.

[0010] In another example, when there are one or more abnormal key points among the multiple second candidate key points, the image processing device may also eliminate the one or more abnormal key points and use the remaining multiple second candidate key points as the multiple first key points.

[0011] In the above design, by verifying multiple first candidate key points, it is possible to correct abnormal key points and supplement missing key points, thereby improving the detection accuracy of key points.

[0012] In one possible design, in order to further determine whether the first target identified by the target detection algorithm is accurate, when the first target is identified incorrectly, the image processing device does not need to perform subsequent processing procedures. Therefore, before verifying multiple first candidate key points, the image processing device can achieve double verification of the first target by determining whether the first code corresponding to the first target is the same as the second code corresponding to the first target. When the first code corresponding to the first target is the same as the second code corresponding to the first target, it means that the first target identified by the target detection algorithm is accurate, and the image processing device can execute the verification process and subsequent processing procedures for multiple first candidate key points. When the first code corresponding to the first target is different from the second code corresponding to the first target, it means that the first target identified by the target detection algorithm is wrong, and the image processing device does not need to perform subsequent processing procedures. This can avoid the waste of subsequent computing resources due to incorrect recognition of the first target. The first code corresponding to the first target is obtained when the image processing device performs target detection on the first image, and the second code corresponding to the first target is determined in the following manner: the image processing first selects the center position point or corner point of the three-dimensional object included in the first target from multiple first candidate key points, and determines the arrangement and combination mode of the three-dimensional object included in the first target according to the center position point or corner point of the three-dimensional object included in the first target, and then the image processing determines the second code corresponding to the first target according to the arrangement and combination mode of the three-dimensional object included in the first target (that is, the structural form of the first target). Each structural form of the marker object (or a certain object) corresponds to a unique code, and the corner point is used to indicate the intersection between multiple contour lines in a certain three-dimensional object.

[0013] In one possible design, the image processing device may determine multiple first candidate key points of the first target in the following manner: after determining that the first image contains the first target, the image processing device first crops the first image according to the bounding box of the first target to obtain a target image containing the first target. Thereafter, the image processing device performs key point detection on the target image to determine multiple first candidate key points of the first target, wherein the bounding box of the first target is obtained by the image processing device performing target detection on the first image.

[0014] In the above design, the area where the first target is located is cropped out for key point detection, so that some useless image information (such as noise information or other invalid information) can be eliminated, so that the key points can be identified more quickly and accurately, and the accuracy of the identified key points is relatively high, and the robustness of key point recognition is also relatively high.

[0015] Optionally, in other possible designs, the image processing device can also determine multiple first candidate key points of the first target by the following implementation method: after determining that the first image contains the first target, the image processing device directly performs key point detection on the first image to determine multiple first candidate key points of the first target. This can reduce the data processing amount of the image processing device, thereby alleviating the data processing pressure of the image processing device.

[0016] On the second aspect, an embodiment of the present application provides a marking object with a three-dimensional structure. When the marking object with a three-dimensional structure (i.e., the first target) includes internal components, the internal components may be composed of a plurality of first three-dimensional objects arranged and combined in a preset arrangement manner. Optionally, a first gap may also exist between the plurality of first three-dimensional objects.

[0017] In this design, by arranging and combining multiple first three-dimensional objects in a preset arrangement to form an internal component, the internal component can have special structural characteristics, and the feature point recognition area of ​​the marking object can be increased, so that even if the marking object is covered with dust (or covered with dust or covered with coal dust) or damaged, the depth and outline of the marking object can be perceived through stereoscopic vision, so that the marking object can be accurately recognized. In addition, when there is a certain gap between the multiple first three-dimensional objects, the gap can be used to enhance the visual depth difference and shadow difference, thereby helping to improve the recognition success rate of the marking object.

[0018] In a possible design, the plurality of second three-dimensional objects may be identical in all structural attributes, or at least two of the plurality of second three-dimensional objects may differ in one or more of the following structural attributes: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing. Exemplarily, taking three first three-dimensional objects as an example, in one example, the three first three-dimensional objects have the same structural attributes and are the same object, such as the three-dimensional shape, front surface shape, and three-dimensional size of the three first three-dimensional objects are the same, and there is no hollowing inside the three first three-dimensional objects or the three first three-dimensional objects all have the same hollowing inside. In another example, two of the three first three-dimensional objects are different in three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing out, for example, the two first three-dimensional objects are different in three-dimensional shape, and one or more of other structural attributes may be the same or different, or two of the three first three-dimensional objects are different in three-dimensional shape and front surface shape, and are the same in other structural attributes (for example, the three-dimensional sizes are the same and there is no hollowing out inside), or two of the three first three-dimensional objects are different in all structural attributes, for example, the two first three-dimensional objects are different in three-dimensional shape, front surface shape and three-dimensional size, and one of the two first three-dimensional objects has a hollowing out inside, while the other first three-dimensional object does not have a hollowing out inside.

[0019] In the above design, different internal components can be formed by changing one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of the object (i.e., whether the object has internal hollowing out) of multiple first three-dimensional objects. This can achieve rapid changes in the structure of the marking object, that is, rapid changes in the ID of the marking object can be achieved, thereby achieving rapid deployment of marking objects with different IDs over a large range and reducing the manufacturing cost of the marking object and the difficulty of algorithm recognition.

[0020] In one possible design, a marking object with a three-dimensional structure may include not only an internal component but also an external component, wherein the external component is composed of a plurality of second three-dimensional objects arranged and combined in a preset arrangement, and the external component is used to surround the internal component. Optionally, a second gap may be provided between the external component and the internal component.

[0021] In the above design, by arranging and combining multiple second three-dimensional objects in a preset arrangement to form an internal component, the external component can have special structural characteristics, so that the external component can be used to distinguish it from the background and assist in the identification of the sign object, so that the sign object can have background distinction, thereby further improving the recognition of the sign object. In addition, the external component surrounds the internal component and can also be used to protect the internal component from damage. In addition, when there is a certain gap between the external component and the internal component, the gap can be used to enhance the visual depth difference and shadow difference, thereby helping to improve the recognition success rate for the sign object.

[0022] In one possible design, through holes may be provided around the external component so that the external component can be transparent from top to bottom, thereby effectively avoiding the accumulation of dust, dirt, coal dust, etc. Optionally, there may be a third gap between the multiple second three-dimensional objects. When there is a certain gap between the multiple second three-dimensional objects, the gap can be used to enhance the visual depth difference and shadow difference, thereby helping to improve the recognition success rate of the marking object.

[0023] In a possible design, the plurality of second three-dimensional objects may be identical in all structural properties, or at least two of the plurality of second three-dimensional objects may differ in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

[0024] In the above design, different external components can be formed by changing one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of the object (i.e., whether the object has internal hollowing). In this way, by randomly combining different external components with different internal components, it is possible to achieve rapid changes in the structure of the marking object, that is, it is possible to achieve rapid changes in the ID of the marking object, so that marking objects with different IDs can be quickly deployed over a large range, and the manufacturing cost and algorithm recognition difficulty of the marking object can be reduced, while helping to improve the flexibility of the application and deployment of the marking object.

[0025] On the third aspect, the embodiments of the present application further provide a marking object with a three-dimensional structure, and the marking object with a three-dimensional structure (i.e., the first target) can also be a three-dimensional marking object with a curved surface, or can also be a three-dimensional marking object with multiple planes, wherein each surface of the marking object with a three-dimensional structure includes a first component, and the first component is composed of a plurality of third three-dimensional objects arranged and combined in a preset arrangement manner, and optionally, a fourth gap may exist between the plurality of third three-dimensional objects.

[0026] In the above design, by arranging and combining a plurality of third three-dimensional objects in a preset arrangement to form the first component, the first component can have special structural characteristics. Afterwards, the first component is used to form a plurality of faces included in the three-dimensional sign object, or is pasted on the plurality of faces included in the three-dimensional sign object by pasting (or other means), so that each face included in the three-dimensional sign object has certain key features, thereby achieving the purpose of observing the three-dimensional sign object from multiple viewing angles, and achieving accurate recognition and detection of the three-dimensional sign object.

[0027] In a possible design, the plurality of third three-dimensional objects may be identical in all structural properties, or at least two of the plurality of third three-dimensional objects may differ in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

[0028] In the above design, different first components can be formed by changing one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of multiple third three-dimensional objects (i.e., whether the object has internal hollowing). In this way, the structure of each face in the three-dimensional sign object can be quickly changed, that is, the ID of each face in the three-dimensional sign object can be quickly changed, thereby achieving accurate identification of the three-dimensional sign object.

[0029] In a possible design, when the sign object with a three-dimensional structure is composed of internal components and a base plate, the base plate is used to carry the internal components, such as the internal components can be fixed to the base plate in a certain fixing manner (such as snap fixing, bolt fixing, latch fixing, magnetic fixing, adhesive fixing, welding fixing, etc.). When the sign object with a three-dimensional structure is composed of internal components, a base plate and external components, the base plate is used to carry the internal components and the external components, such as the internal components and the external components can be fixed to the base plate respectively in the above-mentioned fixing manners for the internal components. When the sign object with a three-dimensional structure is a three-dimensional sign object with an arc surface or a three-dimensional sign object with multiple planes, and each surface of the three-dimensional object includes a first component and a base plate, the first component can be fixed to the base plate in the above-mentioned fixing manners for the internal components. In this way, the design can make the structure of the sign object relatively fixed by using the base plate to support the internal components, the external components or the first components, which helps to improve the stability of the sign object.

[0030] In a fourth aspect, an embodiment of the present application provides a possible image processing device. Exemplarily, the image processing device has a computing device that implements the functions required by the method, or it may also be a functional element (such as a plug-in, component or chip, etc.) that implements the functions required by the method, or it may also be any other device (such as a robot device, a cloud server or a vehicle-mounted device, etc.) that implements the functions required by the method, or it may also be a functional element set in a computing device or any other device. For example, taking the image processing device as a chip set in a computing device as an example, when the image processing device is a chip, the image processing device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists as an input and output interface, and the input and output interface is used for the chip to implement the transceiver of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can implement the reception of the computing device, and the output interface can be used to implement the sending of the computing device. The processor is used to read and execute corresponding computer programs or instructions so that the corresponding functions of the image processing device are implemented.

[0031] When the image processing device is used to implement the function of the image processing device in the method embodiment of the first aspect, the beneficial effects can refer to the description of the image processing device in the first aspect, which will not be repeated here. The image processing device has the function of implementing the behavior of the image processing device in the method example of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the image processing device includes an acquisition module and a processing module. The acquisition module is used to acquire a first image, wherein the first image is obtained by the first sensor shooting the object to be photographed. The processing module is used to perform key point detection on the first image when it is determined that the first target is included in the first image, and determine multiple first key points of the first target. Among them, the first target is a landmark object with a three-dimensional structure, the first target is located on the object to be photographed, the first key point is used to characterize the structural features of the first target, and the multiple first key points are used to determine the internal and external parameter information of the first sensor, or to determine the posture information of the object to be photographed. These modules can perform the corresponding functions in any possible design of the first aspect, and refer to the detailed description in the method example, which will not be repeated here.

[0032] In a fifth aspect, an embodiment of the present application provides a possible image processing device, which includes a transceiver and a processor. Optionally, the image processing device also includes a memory. The memory is used to store computer programs or instructions, and the processor is coupled to the memory and the transceiver. When the processor executes the computer program or instructions, the image processing device executes the method in any possible design of the first aspect above. In one example, the transceiver can be a transceiver device with integrated data transceiver function. In another example, the transceiver can also be composed of a transmitter and a receiver, wherein the transmitter is used to send data and the receiver is used to receive data.

[0033] Exemplarily, taking the image processing device as a chip set in a computing device, when the image processing device is a chip, the communication device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists as an input and output interface, and the input and output interface is used for the chip to realize the transceiver of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can realize the reception of the computing device, and the output interface can be used to realize the sending of the computing device. The processor is used to read and execute the corresponding computer program or instruction, so that the chip realizes the corresponding function of the image processing device.

[0034] In a sixth aspect, an embodiment of the present application provides a possible image processing system, which includes a marker object having a three-dimensional structure, a sensor, and an image processing device, wherein the image processing device can be used to execute the method in any possible design of the first aspect above, the marker object can be used to realize the calibration of the sensor or the positioning of the object to be photographed (the marker object is deployed on the object to be photographed), and the sensor can be used to capture the image information of the marker object or to capture the environmental information of the area where the marker object is located. Exemplarily, the number of the marker objects included in the image processing system can be one or more, the number of the sensors can be one or more, and the number of the image processing devices can be one or more, and the embodiments of the present application do not limit this.

[0035] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the method in any possible design of the first aspect above.

[0036] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a computer, the computer executes a method in any possible design of the first aspect above.

[0037] In the ninth aspect, an embodiment of the present application further provides a chip, which is coupled to a memory, and is used to read a computer program stored in the memory to execute a method in any possible design of the first aspect above.

[0038] In a tenth aspect, an embodiment of the present application further provides a chip system, which includes a processor for supporting a computer device to implement a method in any possible design of the first aspect. In one possible design, the chip system also includes a memory for storing programs and data necessary for the computer device. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0039] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1a A schematic diagram of a possible application scenario provided by an embodiment of the present application is exemplified;

[0041] Figure 1b A schematic diagram of a possible application scenario provided by an embodiment of the present application is exemplified;

[0042] Figure 1c A schematic diagram of a possible application scenario provided by an embodiment of the present application is exemplified;

[0043] Figure 1d A schematic diagram of a possible application scenario provided by an embodiment of the present application is exemplified;

[0044] Figure 2 A schematic diagram exemplarily illustrates a flow chart of an image processing method provided in an embodiment of the present application;

[0045] Figure 3a A possible schematic diagram of internal components provided by an embodiment of the present application is exemplarily shown;

[0046] Figure 3b A possible marking object schematic diagram provided by an embodiment of the present application is exemplarily shown;

[0047] Figure 3c A possible schematic diagram of external components provided by an embodiment of the present application is exemplarily shown;

[0048] Figure 3d An assembly relationship diagram of an internal component and an external component provided by an embodiment of the present application is exemplarily shown;

[0049] Figure 3e Another possible marking object schematic diagram provided by an embodiment of the present application is exemplarily shown;

[0050] Figure 3f Another possible marking object schematic diagram provided by an embodiment of the present application is exemplarily shown;

[0051] Figure 3g Another possible marking object schematic diagram provided by an embodiment of the present application is exemplarily shown;

[0052] Figure 3h Another possible marking object schematic diagram provided by an embodiment of the present application is exemplarily shown;

[0053] Figure 3i A possible schematic diagram of a three-dimensional sign object provided by an embodiment of the present application is exemplarily shown;

[0054] Figure 4 A schematic diagram of a marking object application provided by an embodiment of the present application is exemplarily shown;

[0055] Figure 5 A schematic diagram of key point regression provided by an embodiment of the present application is exemplarily shown;

[0056] Figure 6 A schematic diagram showing a possible structure of an image processing device provided by an embodiment of the present application is exemplified;

[0057] Figure 7 A schematic diagram of the structure of a possible image processing device provided in an embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0058] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0059] The following is an introduction to possible application scenarios to which the image processing method provided by the present application is applicable. It should be noted that these introductions are for the convenience of understanding by those skilled in the art and do not limit the scope of protection claimed by the present application.

[0060] Application scenario 1: multi-camera online calibration.

[0061] Figure 1a The following is a schematic diagram showing a possible application scenario to which the present application embodiment is applicable. Figure 1aAs shown, taking the mark object with a three-dimensional structure as a three-dimensional (3D) marker (i.e., a 3D marker) as an example, five 3D markers with fixed relative positions are deployed in the application scenario 1, such as 3D marker0, 3D marker1, 3D marker2, 3D marker3, and 3D marker4, and three cameras are deployed in the application scenario 1, such as camera 1, camera 2, and camera 3. Among them, the three cameras can simultaneously capture all or part of the five 3D markers. Optionally, when the image processing device obtains the images captured by the three cameras at the same time, it can perform target detection on the images captured by each camera, determine that the images captured by each camera contain targets (such as one or several 3D markers), and identify key points of the targets contained in the images captured by each camera, and obtain multiple key points of the targets contained in the images captured by each camera. Then, the image processing device can determine the external parameter information (or posture information) of each camera based on the multiple key points of the targets contained in the images captured by each camera, so as to realize the online calibration of the external parameters of the three cameras. Optionally, when performing target detection on the image taken by each camera, the image processing device may not only identify the 3D marker contained in the image, but also identify the identification (ID) (or encoding) of the 3D marker contained in the image.

[0062] Optionally, the number of 3D markers deployed in the above application scenario one can be one or more. When the number of 3D markers deployed in the above application scenario one is multiple, it should be ensured that there are 2 or more 3D markers in the shooting field of view of each camera. When the number of 3D markers deployed in the above application scenario one is one, it should be ensured that multiple cameras can capture the 3D marker at the same time.

[0063] Exemplarily, taking camera 1 taking pictures of marker0 and marker1 as an example, after acquiring the image taken by camera 1, the image processing device performs target detection on the image. When it is determined that the image contains two 3D markers (i.e., 3D marker0 and 3D marker1), the two 3D markers contained in the image are identified by key points to obtain multiple key points of the two 3D markers contained in the image. Then, the image processing device can calculate the external parameter information of camera 1 according to the multiple key points of the two 3D markers contained in the image, thereby realizing the online calibration of the external parameters of camera 1. Optionally, when performing target detection on the image taken by camera 1, the image processing device can not only identify the two 3D markers contained in the image, but also identify the IDs of the two 3D markers contained in the image, i.e., the ID of one 3D marker is 0 and the ID of the other 3D marker is 1.

[0064] In addition, it should be noted that this application method can also be applied to the camera intrinsic calibration scenario. In the camera intrinsic calibration scenario, a camera shoots the same 3D marker in different postures to obtain multiple images. Afterwards, for each image, the image processing device performs target detection on the image. When it is determined that the image contains a 3D marker, the image is key-point detected to obtain multiple key points of the 3D marker contained in the image. Then, the image processing device can determine the camera posture information corresponding to the image based on the multiple key points of the 3D marker contained in the image. Finally, the image processing device jointly calculates the camera posture information corresponding to the multiple images, and can determine the intrinsic parameter information of the camera. Exemplarily, the intrinsic parameter information of the camera may include but is not limited to focal length, optical center or distortion parameters.

[0065] Application scenario two: real-time positioning of multiple targets.

[0066] Figure 1b The following is a schematic diagram showing a possible application scenario to which the present application embodiment is applicable. Figure 1bAs shown, taking multiple targets as multiple robots as an example, in the second application scenario, when robots need to be grouped, the location of each robot needs to be identified. At this time, a 3D marker (or multiple 3D markers can be installed) is installed at a visible position of each robot. Afterwards, the image information of the robot is captured by a camera or other sensor (such as a lidar). When the image processing device performs target detection on a captured image and determines that the image contains a 3D marker, the image processing device performs key point detection on the image to determine multiple key points of the 3D marker. Then, the image processing device can determine the spatial posture information of the robot based on the multiple key points of the 3D marker, so as to achieve accurate positioning of each robot, and at the same time, it can be shared through the cloud to achieve the application goal of global positioning of robot grouping.

[0067] For example, taking the application scenario 2 including four robots (such as robot 1, robot 2, robot 3 and robot 4) as an example, when the four robots need to be grouped, a 3D marker can be installed at a visible position of each robot, for example, a 3D marker, such as 3D marker1, is installed at a visible position of robot 1, a 3D marker, such as 3D marker2, is installed at a visible position of robot 2, a 3D marker, such as 3D marker3, is installed at a visible position of robot 3, and a 3D marker, such as 3D marker4, is installed at a visible position of robot 4. When the image processing device performs target detection on a certain image captured by the camera and determines that the image contains 3Dmarker1, the image processing device performs key point detection on the image to determine multiple key points of 3D marker1, and can determine the spatial position information of robot 1 based on the multiple key points of 3D marker1, thereby realizing accurate positioning of robot 1.

[0068] It should be noted that this application method can also be applied in other scenarios, such as grouping multiple autonomous driving vehicles, or grouping unmanned trucks in open-pit mines, or grouping unmanned vehicles in ports and terminals, etc.

[0069] Application scenario three: The robot performs an operation task (such as the robot grabs a target object).

[0070] Figure 1c The following is a schematic diagram showing a possible application scenario to which the present application embodiment is applicable. Figure 1cAs shown, in the application scenario three, a robot (such as robot a), two targets (such as target 1 and target 2) and two cameras (such as camera 0 and camera 1) are deployed. Among them, robot a has visual capabilities, such as robot a has 2D image or 3D image acquisition capabilities, camera 0 is installed on the chassis or mechanical arm of robot a or other visible positions, and camera 1 is installed at any position in the space area where robot a is located. When a 3D marker is needed for positioning and target recognition in the application scenario three, a 3D marker such as 3Dmarker0 can be installed on the chassis or mechanical arm of robot a or other visible positions, a 3D marker such as 3Dmarker1 can be installed on the visible position of target 1, and a 3D marker such as 3D marker2 can be installed on the visible position of target 2.

[0071] For example, taking robot a grasping target object 1 as an example, after acquiring an image (such as image 1) from camera 1, the image processing device can perform target detection on image 1. When it is determined that image 1 contains 3D marker0 and 3D marker1, key point detection can be performed on image 1 to obtain multiple key points of 3D marker0 and multiple key points of 3D marker1. Afterwards, the image processing device can determine the posture information of robot a based on the multiple key points of 3D marker0, and can determine the posture information of target object 1 based on the multiple key points of 3D marker1. Among them, the posture information of robot a and the posture information of target object 1 can be used to guide robot a to grasp target object 1.

[0072] Optionally, in one example, when the image processing device is an independent functional unit embedded in the robot control system or the image processing device is a functional component (such as a chip) encapsulated in the robot control system, the robot control system can generate navigation information of robot a based on the posture information of robot a and the posture information of target object 1, and the navigation information is used to indicate the navigation path from robot a to target object 1.

[0073] In another example, when the image processing device is a computing device or other image processing device or functional component independent of the robot control system, the image processing device can send the position information of robot A and the position information of target object 1 to the robot control system, or when the robot control system obtains the position information of robot A and the position information of target object 1, it can generate navigation information of robot A according to the position information of robot A and the position information of target object 1, and the navigation information is used to indicate the navigation path from robot A to target object 1.

[0074] Optionally, when robot a travels along the navigation path to a preset position of target object 1 (for example, a position 1 meter in front of target object 1), the camera 2 on the robot a's mechanical arm can be used to photograph the spatial area where target object 1 is located. Afterwards, after acquiring an image (for example, image 2) from camera 2, the image processing device can perform target detection on image 2. When it is determined that image 2 contains 3D marker 1, key point detection can be performed on image 2 to obtain multiple key points of 3D marker 1. Then, the image processing device can determine the position and posture information of target object 1 based on the multiple key points of 3D marker 1, so that the grasping point of target object 1 can be accurately determined, so that robot a can accurately use the mechanical arm to grasp target object 1 based on the grasping point of target object 1.

[0075] Application scenario four: multi-sensor calibration and alignment.

[0076] Figure 1d The following is a schematic diagram showing a possible application scenario to which the present application embodiment is applicable. Figure 1d As shown, taking the multi-sensor as a camera (or optical imaging device such as a video camera) and a laser radar (or a depth information acquisition device such as a depth camera) as an example, in order to complete the calibration alignment of the camera (such as a visible light camera) and the laser radar, in this application scenario 4, the spatial positions of the visible light camera and the laser radar are relatively fixed, and a 3D marker is deployed in the shooting field of view of the visible light camera and the laser radar. Among them, the image taken by the visible light camera is a 2D image, and the image taken by the laser radar is a 3D image (or can be called a 3D point cloud map, or can be called a depth map).

[0077] When an image (such as image a) from a visible light camera is acquired, the image processing device performs target detection on image a, and when determining that image a contains a 3D marker, performs key point detection on image a to obtain multiple key points of the 3D marker contained in image a. Afterwards, the image processing device can determine the pose information (or 2D pose) of the visible light camera based on the multiple key points of the 3D marker contained in image a. For example, the image processing device can use the PnP (Perspective-n-Point) algorithm to perform PnP solution on the position coordinates of the multiple key points of the 3D marker contained in image a in the camera coordinate system and the position coordinates of the multiple key points in the world coordinate system to calculate the pose information of the visible light camera.

[0078] When an image from a laser radar (such as image b) is acquired, the image processing device performs target detection on image b, and when it is determined that image b contains a 3D marker, it performs key point detection on image b to obtain multiple key points of the 3D marker contained in image b. Afterwards, the pose information (or 3D pose) of the laser radar can be determined based on the multiple key points of the 3D marker contained in image b. For example, the image processing device can use a PnP algorithm to perform a PnP solution on the position coordinates of the multiple key points of the 3D marker contained in image b in the radar coordinate system and the position coordinates of the multiple key points in the world coordinate system to calculate the pose information of the laser radar.

[0079] Then, the image processing device can solve the relative posture between the visible light camera and the laser radar according to the posture information of the visible light camera, the posture information of the laser radar and the spatial distance relationship between the visible light camera and the laser radar, thereby completing the calibration and alignment of the visible light camera and the laser radar.

[0080] It should be noted that the above application scenarios are only examples. The image processing method provided in this application can also be applied to a variety of other application scenarios (such as identification and positioning of open-pit mine trucks, vehicle positioning in autonomous driving scenarios such as ports and terminals, target positioning in metal-intensive scenarios such as electricity, etc.), and is not limited to the application scenarios exemplified above.

[0081] As described in the background technology, when the existing camera calibration scheme calibrates the camera, the 2D two-dimensional code identification card used cannot be effectively recognized due to dust coverage (or coal dust coverage or dust coverage) or damage, resulting in abnormal image splicing or inaccurate positioning of the object to be photographed. In view of this, the embodiment of the present application provides an image processing method to ensure the effectiveness and accuracy of the identification of the marked object, thereby ensuring the accuracy of the sensor calibration and ensuring the accurate positioning of the object to be photographed.

[0082] The following is based on Figure 1a to Figure 1d The application scenario shown is used to introduce the specific implementation of the image processing method in the embodiment of the present application in detail.

[0083] Figure 2 The following is a flow chart showing an image processing method provided by an embodiment of the present application. Figure 1a to Figure 1dThe application scenario illustrated. The method flow can be executed by an image processing device or a component (such as a chip system or circuit, etc.) that can support the image processing device to implement the functions required by the method. Exemplarily, the image processing device can be a computing device (such as a computer) that has the functions required to implement the method, or it can also be a functional element (such as a plug-in, component or chip, etc.) that has the functions required to implement the method, or it can also be any other device (such as a robot device, a cloud server or a vehicle-mounted device, etc.) that has the functions required to implement the method, or it can also be a functional element set in a computing device or any other device. In order to facilitate the introduction of the technical solution provided in the embodiments of the present application, the following is an introduction taking the image processing device executing the image processing method as an example. Figure 2 As shown, the method includes:

[0084] Step 201: An image processing device acquires a first image.

[0085] Optionally, the first image is obtained by the first sensor photographing (or sensing or capturing) a landmark object with a three-dimensional structure, or the first sensor photographing an object to be photographed. Exemplarily, the first image may be actively sent by the first sensor to the image processing device, or the image processing device actively sends a request to the first sensor to obtain the first image, or the first sensor sends the image processing device according to the user's instructions, or the image processing device sends a request to the first sensor according to the user's instructions, and the embodiments of the present application are not limited to this.

[0086] Among them, a marker object with a three-dimensional structure is deployed on the object to be photographed, for example, a marker object with a three-dimensional structure is deployed at a visible position of the object to be photographed. Optionally, when calibrating the sensor, the marker object with a three-dimensional structure can also be directly used as the object to be photographed without being deployed on a certain object to be photographed. Exemplarily, the first sensor may include but is not limited to: sensors such as image acquisition devices or radar equipment, wherein the image acquisition device may include but is not limited to: visible light cameras, depth cameras (i.e., 3D cameras), fisheye cameras, monocular cameras or depth cameras, etc., and the radar equipment may include but is not limited to: lidar sensors or millimeter wave radar sensors, etc.

[0087] It should be noted that the image obtained when the first sensor captures the object to be captured may contain a landmark object with a three-dimensional structure (ie, the first target), or may not contain a landmark object with a three-dimensional structure.

[0088] In order to facilitate the understanding of sign objects with three-dimensional structures, the design ideas of sign objects with three-dimensional structures are introduced below.

[0089] Based on the imaging principle, it can be known that an image is formed by the change of energy projected onto the imaging element, that is, the change of color or contrast. In the existing sensor calibration scheme, 2D markers (such as checkerboard calibration plates or 2D two-dimensional code identification cards or 2D Apriltag) are used as markers required to assist sensor calibration. However, image recognition of 2D markers is based on the pixel distribution, color, texture and other features of the image. When the 2D marker is affected by factors such as illumination, occlusion, surface dirt (such as dust or coal dust pollution), and damage, the key features in the 2D marker image captured by the sensor will be blocked, resulting in changes in the features of the 2D marker and the inability to effectively identify the 2D marker, which makes it impossible to perform sensor calibration, video stitching or target positioning.

[0090] In view of the above problems, the embodiment of the present application provides a sign object with a three-dimensional structure, which improves the recognition ability of the sign object by constructing visual clues, that is, the perspective projection, occlusion, shadow and other features formed by the three-dimensional shape, front surface shape and structural depth difference of the sign object on the 2D image plane can improve the recognition ability of the sign object in a dirty state, so that the sign object has a relatively good anti-dirt ability and a relatively good anti-dust (or coal dust or dust, etc.) coverage ability. Furthermore, since the contour features of the three-dimensional structure of the sign object are unique and are not affected by changes in surface color and texture features, even if the sign object is covered by dust or dust or coal dust, or is exposed to light or is damaged, the depth and contour of the sign object can still be perceived by using stereoscopic vision, so that the sign object can be identified, so that the interference of dust or dust or coal dust or light or damage can be effectively ignored.

[0091] In a possible implementation, the sign object with a three-dimensional structure includes an internal component. The internal component may be composed of a plurality of first three-dimensional objects arranged and combined in a preset arrangement (such as a four-grid arrangement or a nine-grid arrangement or other arrangement). Exemplarily, the first three-dimensional object may include but is not limited to a cylinder, a cube, a cuboid, a cylinder, a sphere, a prism, a pyramid, a cone, a prism, a truncated cone, a trapezoid (such as a trapezoidal column), a fan (such as a fan-shaped column), an arcuate (such as an arcuate column), a triangular pyramid, a polygonal column (such as a triangular column, a quadrangular column or a pentagonal column, etc.) or other objects such as polyhedrons with a three-dimensional structure. Optionally, the sign object with a three-dimensional structure may also include a base plate, wherein the base plate is used to carry a plurality of first three-dimensional objects.

[0092] Among them, there may be a first gap between multiple first three-dimensional objects. For example, the first gap may be 0, or may be greater than 0. When the first gap is 0, there is no gap between multiple first three-dimensional objects. When the first gap is a certain value greater than 0 (such as 1 cm), there is a gap of this numerical distance between multiple first three-dimensional objects. In addition, the gaps of a certain distance between multiple first three-dimensional objects can be used to enhance the visual depth difference and shadow difference, thereby helping to improve the recognition success rate for the sign object. Exemplarily, taking multiple different first three-dimensional objects as an example, multiple different first three-dimensional objects can be arranged and transformed according to a preset arrangement to form different internal components.

[0093] Optionally, the arrangement and combination of multiple first three-dimensional objects (which can also be understood as a coding method) can be used to distinguish different marking objects, while allowing the marking object to have unique internal features (i.e., a unique ID (or can be called a code)), which helps to realize the recognition of the marking object (or the ID recognition of the marking object) and helps to realize redundant calculation of features, thereby ensuring that when some information of the marking object is missing, the recognition success rate of the marking object is high.

[0094] In addition, for internal components, in order to improve the feature difference, the surface of the internal components can also be processed accordingly. For example, the surface of the internal components can be processed by surface treatment, surface painting or surface coating, so as to improve the visual clue characteristics of the internal components, that is, to increase visual information such as color, texture, contrast, etc. Optionally, light-emitting devices, such as light-emitting diodes (LEDs), can also be installed locally on the internal components to increase visual information such as color and contrast. For example, installing LEDs at key structural points of internal components (such as fixed points or intersections, etc.) can effectively improve the recognition ability (or recognition ability) of the sign object in a dark environment, thereby increasing the scope of use of the sign object. Optionally, the LED installed on the internal component can outline the main outline of the sign object in a uniform light-emitting manner or other light-emitting manners, forming a self-luminous positioning mark, which can effectively deal with unclear recognition of the sign object surface caused by changes in lighting, rain, fog, snow, etc., as well as dirt. Furthermore, active detail enhancement can also be achieved by adding a background board of the internal component or the sign object.

[0095] It should be noted that the internal components formed by arranging and combining multiple first three-dimensional objects in a preset arrangement can be a plastic structure, or a metal structure, or other structures. Among them, the internal components or sign objects (such as sign objects composed of a base plate and internal components) can be produced by 3D printing, or can also be produced by metal processing, or can also be produced by other processing methods. Exemplarily, for internal components, the internal components can adopt a full black design, which can effectively avoid the failure of color and texture due to obstruction such as coal dust.

[0096] In one example, multiple first three-dimensional objects may be the same object (i.e., an object having the same three-dimensional shape, front surface shape, three-dimensional size, and internal structure (i.e., whether the object has internal hollowing)). For example, the same object is a cube-shaped object. The multiple identical cube objects may be arranged and combined in a random arrangement to form an internal component, or the multiple identical cube objects may be arranged and combined in other arrangements to form an internal component. For example, taking four identical first three-dimensional objects, with a preset arrangement of a four-square grid, the four identical first three-dimensional objects may be randomly arranged in the four-square grid positions to form an internal component.

[0097] In another example, at least two of the plurality of first three-dimensional objects are different objects. Optionally, the at least two first three-dimensional objects can form different objects by changing one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing. Among them, at least two of the multiple first three-dimensional objects have different three-dimensional shapes (i.e., the overall shapes of the first three-dimensional objects) (for example, there are three first three-dimensional objects, one of which is a cylinder, another of which is a cube, and the remaining one is also a cube), and / or, at least two of the multiple first three-dimensional objects have different front surface shapes (for example, there are three first three-dimensional objects, one of which is a square, another of which is a rectangle, and the remaining one is also a rectangle), and / or, at least two of the multiple first three-dimensional objects have different three-dimensional sizes (i.e., the overall sizes of the first three-dimensional objects) (for example, there are three first three-dimensional objects, one of which has a three-dimensional size of 10, another of which has a three-dimensional size of 20, and the remaining one is also 20), and / or, one or more of the multiple first three-dimensional objects have internal hollowing (for example, a surface (such as the front surface) of a first three-dimensional object has a groove or a hole). The front surface of the first solid object may form a depth difference (or may be called a height difference) with the back plate to facilitate the formation of light and shade and shadows, and the depth difference is used to miss dust or dirt or coal dust, etc. Exemplarily, the shape of the front surface of the first solid object may include but is not limited to a circle or a square (such as a square, a rectangle or other squares), etc. Exemplarily, the shape of the hollow (such as a groove or a hole) may include but is not limited to a circle, a square (such as a square or a rectangle or a rectangle), a triangle, a fan, a trapezoid, a quadrilateral or other polygons, etc.

[0098] For example, taking four first three-dimensional objects (such as three-dimensional object a1, three-dimensional object a2, three-dimensional object a3 and three-dimensional object a4), and the three-dimensional sizes and three-dimensional shapes of the four first three-dimensional objects are the same (for example, the three-dimensional shapes of the four first three-dimensional objects are all cubes, and the sizes of the four first three-dimensional objects are the same), the front surface shape of three-dimensional object a1, the front surface shape of three-dimensional object a2 and the front surface shape of three-dimensional object a3 are the same, all are squares, but there is a circular hole in the middle part of the front surface of three-dimensional object a2, there is a square hole in the middle part of the front surface of three-dimensional object a1 and the middle part of the front surface of three-dimensional object a3, and the front surface shape of three-dimensional object a4 is also a square, but there is neither a circular hole nor a square hole in the middle part of the front surface of three-dimensional object a4. Afterwards, the four first three-dimensional objects can be arranged and combined according to a preset arrangement mode (i.e., a layout mode, also referred to as an encoding mode). For example, the four first three-dimensional objects can be arranged and combined according to a four-square arrangement mode. In this way, the four first three-dimensional objects can be arranged and combined in various forms at the four-square positions, thereby forming internal components of various structural forms. Each structural form of the internal component corresponds to a unique ID.

[0099] For example, a possible arrangement combination is: the three-dimensional object a1 is arranged in the second grid position of the first row of the four-square grid, the three-dimensional object a2 is arranged in the first grid position of the first row of the four-square grid, the three-dimensional object a3 is arranged in the second grid position of the second row of the four-square grid, and the three-dimensional object a4 is arranged in the first grid position of the second row of the four-square grid. In this way, the following can be formed: Figure 3a The internal components shown. Among them, the four first three-dimensional objects are arranged and combined to form Figure 3a The structure of the internal components shown is unique, making Figure 3a The internal components shown correspond to a unique ID. In one example, the internal components themselves can be used as a sign object. In another example, when the sign object also includes a base plate, the internal components formed by the four three-dimensional objects are combined with the base plate to form a sign object such as Figure 3b The logo object shown (ie, the logo object includes internal components and a base plate).

[0100] For another example, continue to take the case where there are four first three-dimensional objects (such as three-dimensional object a1, three-dimensional object a2, three-dimensional object a3 and three-dimensional object a4), and the three-dimensional size and three-dimensional shape of the four first three-dimensional objects are the same (for example, the three-dimensional shape of the four first three-dimensional objects is a cube, and the size of the four first three-dimensional objects is the same), the front surface shape of the three-dimensional object a1, the front surface shape of the three-dimensional object a2 and the front surface shape of the three-dimensional object a3 are the same, all of which are squares, but there is a groove in the middle part of the front surface of the three-dimensional object a1, there is a circular hole in the middle part of the front surface of the three-dimensional object a3, and there is neither a groove nor a circular hole in the middle part of the front surface of the three-dimensional object a2 and the middle part of the front surface of the three-dimensional object a4. Afterwards, the four first three-dimensional objects can be arranged and combined according to a preset arrangement method, for example, the four first three-dimensional objects are arranged and combined according to a four-square arrangement method, so that the four first three-dimensional objects can be arranged and combined in various forms at the four-square positions, so as to form internal components of various structural forms. Among them, each structural form of internal components corresponds to a unique ID.

[0101] For example, a possible arrangement combination is: the three-dimensional object a3 is arranged in the second grid position of the first row of the four-grid, the three-dimensional object a1 is arranged in the first grid position of the first row of the four-grid, the three-dimensional object a4 is arranged in the second grid position of the second row of the four-grid, and the three-dimensional object a2 is arranged in the first grid position of the second row of the four-grid, forming a new internal component. Among them, the structure of the new internal component formed by the arrangement and combination of the four first three-dimensional objects is also unique, so that the new internal component also corresponds to a unique ID. In one example, the new internal component itself can be used as a marking object. In another example, when the marking object also includes a base plate, the new internal component formed by the four three-dimensional objects is combined with the base plate to form a new marking object (that is, the new marking object includes a new internal component and a base plate). Among them, the internal component can be fixed on the base plate in a certain fixing manner (such as a snap-on fixing manner, a bolt fixing manner, a pin fixing manner, a magnetic fixing manner, a pasting fixing manner, or a welding fixing manner, etc.), so that the internal component is combined with the base plate.

[0102] For another example, continue to take the case where there are four first three-dimensional objects (such as three-dimensional object a1, three-dimensional object a2, three-dimensional object a3 and three-dimensional object a4), and the three-dimensional size and three-dimensional shape of the four first three-dimensional objects are the same (for example, the three-dimensional shape of the four first three-dimensional objects is a cube, and the size of the four first three-dimensional objects is the same), the front surface shape of the three-dimensional object a1, the front surface shape of the three-dimensional object a2 and the front surface shape of the three-dimensional object a3 are the same, all of which are squares, but there is a groove in the middle part of the front surface of the three-dimensional object a1, there is a square hole in the middle part of the front surface of the three-dimensional object a2, there is a circular hole in the middle part of the front surface of the three-dimensional object a3, and there is no groove, square hole or circular hole in the middle part of the front surface of the three-dimensional object a4. Afterwards, the four first three-dimensional objects can be arranged and combined according to a preset arrangement method, for example, the four first three-dimensional objects are arranged and combined according to a four-square arrangement method, so that the four first three-dimensional objects can be arranged and combined in various forms at the four-square positions, so as to form internal components of various structural forms. Each internal component of the structural form corresponds to a unique ID.

[0103] For example, a possible arrangement combination is: the three-dimensional object a4 is arranged in the second grid position of the first row of the four-grid, the three-dimensional object a2 is arranged in the first grid position of the first row of the four-grid, the three-dimensional object a1 is arranged in the second grid position of the second row of the four-grid, and the three-dimensional object a3 is arranged in the first grid position of the second row of the four-grid, forming a new internal component. Among them, the structure of the new internal component formed by the arrangement and combination of the four first three-dimensional objects is also unique, so that the new internal component also corresponds to a unique ID. In one example, the new internal component itself can be used as a marking object. In another example, when the marking object also includes a base plate, the new internal component formed by the four three-dimensional objects is combined with the base plate to form a new marking object (that is, the new marking object includes the new internal component and the base plate).

[0104] It should be noted that the above content is only an illustrative introduction to the structural forms of several internal components. The above-mentioned multiple first three-dimensional objects (for example, 4 three-dimensional objects) change according to one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of the object, and the multiple changed first three-dimensional objects are arranged and combined according to a preset arrangement method, which can form internal components of various structural forms, which are not listed one by one in the embodiments of the present application.

[0105] In another possible implementation, the marking object with a three-dimensional structure includes not only the above-mentioned internal components but also an external component. The external component is composed of a plurality of second three-dimensional objects arranged and combined in a preset arrangement (such as a circular ring or a square ring (such as a square ring or a rectangular ring) or an arrangement of other shaped rings), and the external component is used to surround the internal component. Exemplarily, the second three-dimensional object may include but is not limited to a cylinder, a cube, a cuboid, a cylinder, a sphere, a prism, a pyramid, a cone, a prism, a truncated cone, a trapezoid (such as a trapezoidal column), a fan (such as a fan-shaped column), an arcuate (such as an arcuate column), a triangular pyramid, a polygonal column (such as a triangular column, a quadrangular column or a pentagonal column, etc.) or other objects such as polyhedrons with a three-dimensional structure.

[0106] Among them, there may be a second gap between the external component and the internal component. For example, the second gap may be 0, or may be greater than 0. When the second gap is 0, there is no gap between the external component and the internal component. When the second gap is a certain value greater than 0 (such as 1 cm), there is a gap of this numerical distance between the external component and the internal component. In addition, a gap of a certain distance between the external component and the internal component can be used to enhance the visual depth difference and shadow difference, thereby helping to improve the recognition success rate for the sign object. Exemplarily, taking a plurality of different second three-dimensional objects as an example, a plurality of different second three-dimensional objects can be arranged and transformed according to a preset arrangement to form different external components.

[0107] Optionally, the arrangement and combination of multiple second three-dimensional objects can be combined with the arrangement and combination of multiple first three-dimensional objects to form a unified ID for distinguishing different marking objects, while enabling the marking objects to have unique internal features (i.e., unique IDs), which helps to realize ID recognition of the marking objects and helps to realize redundant calculation of features, thereby ensuring a high recognition success rate for the marking objects when some information of the marking objects is missing.

[0108] Exemplarily, there may be a third gap between the multiple second three-dimensional objects. For example, there may be a certain gap between the multiple second three-dimensional objects, or there may be no gap. When there is a certain gap between the multiple second three-dimensional objects, the gap between the multiple second objects can be used to enhance the visual depth difference and shadow difference, which helps to improve the ability of the marking object to resist dirt or dust (or dust or coal dust, etc.), thereby enhancing the environmental adaptability of the marking object, making the marking object not only suitable for environments that are prone to dirt such as coal mines, but also suitable for indoor and outdoor scenes with good environments, and the recognition success rate of the marking object is further improved.

[0109] Optionally, through holes may be provided around the external component, so that the external component can be transparent from top to bottom, thereby effectively avoiding the accumulation of dust, dirt, coal dust, etc.

[0110] Exemplarily, when the marker object includes internal components and external components, the structural design of the internal components can make the internal components have special structural characteristics and can increase the feature point recognition area to a certain extent, so that the recognition of the marker object can be further improved, thereby improving the recognition accuracy and robustness of the marker object (or ID), and helping to improve the positioning accuracy of key points in the marker object, while the external components can be used to distinguish from the background and assist in the recognition of the marker object, so that the marker object has background distinction, thereby further improving the recognition accuracy of the marker object. In addition, the external components can also be used to protect the internal components from damage. Among them, changes in one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of the object (i.e., whether the object has internal hollowing) of the multiple second three-dimensional objects included in the external component can make the external component formed by the arrangement and combination of the multiple second three-dimensional objects have background distinguishability (for example, the edge has some special textures (such as wavy textures) and structures), directionality (for example, through the special design of the edges around, it is easy to judge the positive direction of the marked object), ID information (such as edge structure coding), structural height change (for example, a decreasing height change from the upper right corner to the lower left corner of the external component) and other characteristics.

[0111] In addition, for external components, in order to improve the feature difference, the surface of the external components can also be processed accordingly. For example, the surface of the external components can be processed by surface treatment, surface painting or surface coating, so as to improve the visual clue characteristics of the external components, that is, to increase visual information such as color, texture, contrast, etc. Optionally, light-emitting devices, such as LEDs, can also be installed locally on the external components to increase visual information such as color and contrast. For example, installing LEDs at key structural points of the external components (such as fixed points or intersections, etc.) can effectively improve the recognition ability of the sign object in a dark environment, thereby increasing the scope of use of the sign object. Optionally, the LED installed on the external component can outline the main outline of the sign object in a uniform light-emitting manner or other light-emitting manners, forming a self-luminous positioning mark, which can effectively deal with unclear recognition of the sign object surface caused by changes in lighting, rain, fog, snow, etc., as well as dirt. Furthermore, active detail enhancement can be achieved by adding a background board of the external component or the sign object.

[0112] It should be noted that the external component formed by arranging and combining multiple second three-dimensional objects in a preset arrangement can be a plastic structure, or a metal structure, or other structures. Among them, the external component or the sign object (such as a sign object composed of an internal component, a base plate and an external component) can be produced by 3D printing, or can also be produced by metal processing, or can also be produced by other processing methods. Exemplarily, for a sign object including an internal component and an external component, both the external component and the internal component in the sign object can be designed in full black, which can effectively avoid the failure of color and texture due to obstruction such as coal dust.

[0113] In one example, multiple second three-dimensional objects can be the same object (i.e., objects with the same three-dimensional shape, front surface shape, three-dimensional size, and internal structure of the object). For example, the same object is an object with a three-dimensional shape of a rectangular parallelepiped. The multiple identical rectangular parallelepiped objects can be randomly arranged at multiple positions of the square ring according to the arrangement of the square ring to surround and form an external component, or the multiple identical rectangular parallelepiped objects can also be arranged and combined in other arrangements to form an external component. Afterwards, the external component can be combined with the bottom plate and the internal component of any structural form formed by the above-mentioned multiple first three-dimensional objects to form a sign object with a three-dimensional structure. Wherein, when the sign object includes a bottom plate, the bottom plate is used to carry the internal component and the external component. For example, taking 6 identical second three-dimensional objects, the preset arrangement method is the arrangement method of the square ring as an example, the 6 identical second three-dimensional objects can be randomly arranged at 6 positions of the square ring to form an external component. Among them, the external components and the internal components can be fixed on the base plate by some fixing method (such as snap fixing, bolt fixing, latch fixing, magnetic fixing, adhesive fixing, welding fixing, etc.), so that the external components, the internal components and the base plate are combined together.

[0114] In another example, at least two of the plurality of second three-dimensional objects are different objects. Optionally, the at least two second three-dimensional objects can form different objects by changing one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing. Among them, at least two of the multiple second three-dimensional objects have different three-dimensional shapes (i.e., the overall shapes of the second three-dimensional objects) (for example, there are five second three-dimensional objects, of which three have three-dimensional shapes of rectangular parallelepipeds, and the remaining two have three-dimensional shapes of cubes), and / or, at least two of the multiple second three-dimensional objects have different front surface shapes (for example, there are five second three-dimensional objects, of which three have front surface shapes of rectangulars, and the remaining two have front surface shapes of squares), and / or, at least two of the multiple second three-dimensional objects have different three-dimensional sizes (i.e., the overall sizes of the second three-dimensional objects) (for example, there are five second three-dimensional objects, of which three have three-dimensional sizes of 20, and the remaining two have three-dimensional sizes of 10), and / or, one or more of the multiple second three-dimensional objects have internal hollowing (for example, a groove or hole exists on a surface (such as the front surface) of a second three-dimensional object). Optionally, when the arrangement and combination of the plurality of second three-dimensional objects form an external component with a depth difference, the external component may have different depths and light-dark contrasts.

[0115] For example, taking 19 second three-dimensional objects (such as three-dimensional object b1, three-dimensional object b2, three-dimensional object b3, three-dimensional object b4, three-dimensional object b5, three-dimensional object b6, three-dimensional object b7, three-dimensional object b8, three-dimensional object b9, three-dimensional object b10, three-dimensional object b11, three-dimensional object b12, three-dimensional object b13, three-dimensional object b14, three-dimensional object b15, three-dimensional object b16, three-dimensional object b17, three-dimensional object b18 and three-dimensional object b19) as an example, among which 7 second three-dimensional objects (such as three-dimensional object b1, three-dimensional object b2 The three-dimensional shapes of the second three-dimensional objects (such as the three-dimensional object b8, the three-dimensional object b9, the three-dimensional object b10, the three-dimensional object b11, the three-dimensional object b12, the three-dimensional object b13, the three-dimensional object b14, and the three-dimensional object b15) are all trapezoidal, and the three-dimensional shapes of the other eight second three-dimensional objects (such as the three-dimensional object b8, the three-dimensional object b9, the three-dimensional object b10, the three-dimensional object b11, the three-dimensional object b12, the three-dimensional object b13, the three-dimensional object b14, and the three-dimensional object b15) are all rectangular parallelepiped, and the three-dimensional shapes of the remaining four second three-dimensional objects (such as the three-dimensional object b16, the three-dimensional object b17, the three-dimensional object b18, and the three-dimensional object b19) are all cubes. Among them, the front surface shapes of the three-dimensional objects b16, the three-dimensional objects b17, the three-dimensional objects b18, and the three-dimensional objects b19 are all square, and there are circular holes in the middle parts of the front surfaces of the three-dimensional objects b16, the three-dimensional objects b17, the three-dimensional objects b18, and the three-dimensional objects b19. In addition, the front surface shapes of the three-dimensional objects b1, b2, b3, b4, b5, b6, b7, b8, b9, b10, b11, b12, b13, b14 and b15 are all rectangular. Afterwards, the 19 second three-dimensional objects can be arranged and combined according to a preset arrangement. For example, the 19 second three-dimensional objects are arranged and combined in a square ring arrangement, and the surrounding arrangement forms an external component. In this way, the 19 second three-dimensional objects can be arranged and combined in various forms at various positions of the square ring, so as to form external components of various structural forms. Among them, each structural form of the external component corresponds to a unique ID.

[0116] For example, a possible arrangement combination is: three-dimensional object b16, three-dimensional object b17, three-dimensional object b18 and three-dimensional object b19 are arranged at the four corners of the square ring, three-dimensional object b1, three-dimensional object b2 and three-dimensional object b3 can be arranged at three positions selected from the first vertical side of the square ring, three-dimensional object b4, three-dimensional object b5, three-dimensional object b6 and three-dimensional object b7 can be arranged at four positions selected from the second vertical side of the square ring, three-dimensional object b8, three-dimensional object b9, three-dimensional object b10 and three-dimensional object b11 can be arranged at four positions selected from the first horizontal side of the square ring, three-dimensional object b12, three-dimensional object b13, three-dimensional object b14, three-dimensional object b15 can be arranged at four positions selected from the second horizontal side of the square ring, so as to form the following Figure 3c The external components shown in FIG. Among them, the 19 second three-dimensional objects are arranged and combined to form Figure 3c The structure of the external components shown is also unique, making Figure 3c The external component shown also corresponds to a unique ID. In one example, when the marker object includes an internal component (such as an internal component formed by four first three-dimensional objects) and an external component (such as Figure 3c ) when the internal components formed by the four first three-dimensional objects and Figure 3c The external components shown can be combined to form a sign object with a three-dimensional structure. In another example, when the sign object includes an internal component (such as an internal component formed by four first three-dimensional objects), a base plate and an external component (such as Figure 3c When the external component shown in FIG. 1 is formed, the internal component, the bottom plate and the Figure 3c The external components shown can be Figure 3d The assembly relationships shown are combined to form a sign object with a three-dimensional structure.

[0117] For another example, continue to take 19 second three-dimensional objects (such as three-dimensional object b1, three-dimensional object b2, three-dimensional object b3, three-dimensional object b4, three-dimensional object b5, three-dimensional object b6, three-dimensional object b7, three-dimensional object b8, three-dimensional object b9, three-dimensional object b10, three-dimensional object b11, three-dimensional object b12, three-dimensional object b13, three-dimensional object b14, three-dimensional object b15, three-dimensional object b16, three-dimensional object b17, three-dimensional object b18 and three-dimensional object b19) as an example, among which 7 second three-dimensional objects (such as three-dimensional object b1, three-dimensional object b2, three-dimensional object b3, three-dimensional object b4, three-dimensional object b5, three-dimensional object b6, three-dimensional object b7, three-dimensional object b8, three-dimensional object b9, three-dimensional object b10, three-dimensional object b11, three-dimensional object b12, three-dimensional object b13, three-dimensional object b14, three-dimensional object b15, three-dimensional object b16, three-dimensional object b17, three-dimensional object b18 and three-dimensional object b19) are used as an example. 2. The three-dimensional shapes of the three-dimensional objects b3, b4, b5, b6, and b7 are all cylinders, and the three-dimensional shapes of the other eight second three-dimensional objects (such as the three-dimensional object b8, b9, b10, b11, b12, b13, b14, and b15) are all cubes, and the three-dimensional shapes of the remaining four second three-dimensional objects (such as the three-dimensional object b16, b17, b18, and b19) are all triangular prisms. Among them, the front surface shapes of the three-dimensional objects b1, b2, b3, b4, b5, b6, and b7 are all circular, and the front surface shapes of the three-dimensional objects b16, b17, b18, and b19 are all triangular. In addition, the front surface shapes of the three-dimensional objects b8, b9, b10, b11, b12, b13, b14 and b15 are all square, and there are circular holes in the middle parts of the front surfaces of the three-dimensional objects b8, b9, b10, b11, b12, b13, b14 and b15. Afterwards, the 19 second three-dimensional objects can be arranged and combined according to a preset arrangement method. For example, the 19 second three-dimensional objects are arranged and combined according to the arrangement method of a square ring, and the surrounding arrangement forms an external component. In this way, the 19 second three-dimensional objects can be arranged and combined in various forms at various positions of the square ring, so as to form external components of various structural forms. Among them, each structural form of the external component corresponds to a unique ID.

[0118] For example, a possible arrangement combination is: three-dimensional objects b16, b17, b18 and b19 are arranged at the four corners of a square ring, three-dimensional objects b1, b2 and b3 can be arranged at three positions selected from the first horizontal side of the square ring, three-dimensional objects b4, b5, b6 and b7 can be arranged at three positions selected from the first vertical side of the square ring, three-dimensional objects b8, b9, b10 and b11 can be arranged at four positions selected from the second vertical side of the square ring, and three-dimensional objects b12, b13, b14 and b15 can be arranged at four positions selected from the second horizontal side of the square ring to form a new external component. The structure of the new external component formed by the arrangement and combination of the 19 second three-dimensional objects is also unique, so that the new external component also corresponds to a unique ID. In one example, when the sign object includes an internal component (such as an internal component formed by four first three-dimensional objects) and the new external component, the internal component formed by four first three-dimensional objects and the new external component can be combined to form a sign object with a three-dimensional structure. In another example, when the sign object includes an internal component (such as an internal component formed by four first three-dimensional objects), a base plate, and the new external component, the internal component formed by four first three-dimensional objects, the base plate, and the new external component can also be combined as follows: Figure 3d The assembly relationships shown are combined to form a sign object with a three-dimensional structure.

[0119] It should be noted that the above content is only an illustrative introduction to the structural forms of several external components. The above-mentioned multiple second three-dimensional objects (for example, 19 three-dimensional objects) change according to one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of the object, and the multiple second three-dimensional objects after the change are arranged and combined according to a preset arrangement method, which can form external components of various structural forms, which are not listed one by one in the embodiments of the present application.

[0120] Optionally, when the internal components of the above-mentioned various structural forms are randomly combined (or arbitrarily combined) with the external components of the above-mentioned various structural forms, marking objects of various structural forms can be formed. In this way, through this combination method, the rapid change of the marking object structure can be realized, that is, the rapid change of the marking object ID can be realized, so that marking objects with different IDs can be quickly deployed in a large range, and the manufacturing cost and algorithm recognition difficulty of the marking object can be reduced, and at the same time, it helps to improve the flexibility of marking object application and deployment. Among them, each structural form of the marking object will correspond to an ID.

[0121] For example, the internal components formed by the four first three-dimensional objects are as described above. Figure 3a Taking the internal components shown in the figure as an example, when the logo object also includes a base plate, the above Figure 3a The internal components shown are the same as the base plate, Figure 3c The external components shown can be combined to form Figure 3e The flag object shown. Among them, Figure 3a The arrangement of the four first three-dimensional objects included in the internal component shown forms a certain depth difference, which forms a certain shadow difference, and among the four first three-dimensional objects, one first three-dimensional object has a circular hole structure, and two first three-dimensional objects have a square hole structure, and the circular hole structure and the square hole structure form obvious edge and contour changes; Figure 3c The arrangement of the 19 second three-dimensional objects included in the external component shown forms a certain depth difference, and there are circular hollow structures in the middle parts of the front surfaces of 4 of the 19 second three-dimensional objects, and the circular hollow structures form obvious edge and contour changes.

[0122] For another example, taking an internal component formed by nine first three-dimensional objects as internal component a and an external component formed by a plurality of second three-dimensional objects as external component a, the internal component a and the external component a are combined together to form the following Figure 3f The flag object shown. Optional. Figure 3f The illustrated sign object may also include a base plate, which is combined with the internal component a and the external component a to form a corresponding sign object. The nine first three-dimensional objects included in the internal component a are arranged in a concave-convex manner in the nine-square grid position and are not in the same plane. The arrangement of the multiple second three-dimensional objects included in the external component a forms a certain depth difference, and the depth difference forms a certain shadow difference.

[0123] For another example, taking an internal component formed by nine first three-dimensional objects as internal component b and an external component formed by a plurality of second three-dimensional objects as external component b, the internal component b and the external component b are combined together to form the following: Figure 3g The flag object shown. Optional. Figure 3g The illustrated sign object may also include a base plate, which is combined with the internal component b and the external component b to form a corresponding sign object. The nine first three-dimensional objects included in the internal component b have a hollow structure (such as a central hole or groove), and the hollow structure forms obvious edge and contour changes; the arrangement of the multiple second three-dimensional objects included in the external component b forms a certain groove, and the arrangement of the multiple second three-dimensional objects included in the external component b forms a certain depth difference, and the groove and the depth difference form a relatively obvious shadow difference.

[0124] For another example, taking an internal component formed by nine first three-dimensional objects as an internal component c and an external component formed by a plurality of second three-dimensional objects as an external component c, the internal component c and the external component c are combined together to form the following Figure 3h The flag object shown. Optional. Figure 3h The illustrated sign object may also include a base plate, which is combined with the internal component c and the external component c to form a corresponding sign object. The nine first three-dimensional objects included in the internal component c have a hollow structure (such as a central hole and a groove), and the hollow structure forms obvious edge and contour changes; the arrangement of the multiple second three-dimensional objects included in the external component b forms a certain depth difference, and the depth difference forms a certain shadow difference.

[0125] In another possible implementation, the sign object with a three-dimensional structure may also be a three-dimensional sign object with an arc surface (such as a circular arc surface or a fan-shaped arc surface, etc.), or may also be a three-dimensional sign object with multiple planes. Optionally, each surface included in the sign object with a three-dimensional structure includes a first component. Among them, the first component may be composed of a plurality of third three-dimensional objects arranged and combined in a preset arrangement (such as a four-square arrangement or a nine-square arrangement or other arrangement). In one example, the first component may be directly used as a surface (such as an arc surface or a plane) of a three-dimensional sign object. In another example, the first component may also be pasted on the surface included in the three-dimensional sign object, or the first component may also be fixed on the surface included in the three-dimensional sign object in other ways. In this way, the method can achieve the purpose of fixing key features on multiple surfaces of the three-dimensional sign object, thereby achieving the purpose of observing the three-dimensional sign object from multiple perspectives, and can achieve accurate recognition and detection of the three-dimensional sign object. Exemplarily, the third three-dimensional object may include but is not limited to a cylinder, a cube, a cuboid, a cylinder, a sphere, a prism, a pyramid, a cone, a prism, a frustum, a truncated cone, a trapezoidal body (such as a trapezoidal column), a fan-shaped body (such as a fan-shaped column), an arcuate body (such as an arcuate column), a triangular pyramid, a polygonal column (such as a triangular column, a quadrangular column or a pentagonal column, etc.) or other polyhedrons with a three-dimensional structure.

[0126] Optionally, when the three-dimensional sign object includes a base plate, the first component can be fixed on the base plate, the base plate is used to carry the first component, the base plate and the first component directly constitute a surface of the three-dimensional sign object, or, after the base plate and the first component are combined, they are fixed to the surface included in the three-dimensional sign object by pasting or other methods. Among them, the first component can be fixed to the base plate by a certain fixing method (such as snap fixing, bolt fixing, latch fixing, magnetic fixing, pasting fixing, welding fixing, etc.), so that the first component is combined with the base plate.

[0127] Among them, there may be a fourth gap between multiple third three-dimensional objects. For example, the fourth gap may be 0, or may be greater than 0. When the fourth gap is 0, there is no gap between multiple third three-dimensional objects. When the fourth gap is a certain value greater than 0 (such as 1 cm), there is a gap of this numerical distance between multiple third three-dimensional objects. In addition, the gaps of a certain distance between multiple third three-dimensional objects can be used to enhance the visual depth difference and shadow difference, thereby helping to improve the recognition success rate of three-dimensional sign objects. Exemplarily, taking multiple different third three-dimensional objects as an example, multiple different third three-dimensional objects can be arranged and transformed according to a preset arrangement to form different first components. The different first components can be used to constitute multiple faces of a three-dimensional sign object, or can also be fixed on multiple faces of a three-dimensional sign object by pasting or other methods.

[0128] Optionally, the arrangement and combination of multiple third three-dimensional objects can be used to distinguish multiple faces of a three-dimensional sign object, and at the same time, the multiple faces of the three-dimensional sign object can have unique IDs. When a sensor (such as a camera) captures the three-dimensional sign object, the IDs (or codes) and posture information of the multiple faces in the three-dimensional sign object can be identified, so that the three-dimensional sign object can be accurately identified and detected through the known position coordinates and sizes of each face of the three-dimensional sign object, and the sensor can be accurately calibrated.

[0129] It should be noted that the first component formed by arranging and combining multiple third three-dimensional objects in a preset arrangement can be a plastic structure, or a metal structure, or other structures. Among them, the three-dimensional sign object (such as the three-dimensional sign object composed of the first component) or the first component can be produced by 3D printing, or can also be produced by metal processing, or can also be produced by other processing methods.

[0130] In one example, multiple third three-dimensional objects may be the same object (i.e., an object having the same three-dimensional shape, front surface shape, three-dimensional size, and internal structure (i.e., whether the object has internal hollowing)). For example, the same object is a cube-shaped object. The multiple identical cube-shaped objects may be arranged and combined in a random arrangement to form a first component, or the multiple identical cube-shaped objects may be arranged and combined in other arrangements to form a first component. For example, taking four identical third three-dimensional objects, the preset arrangement is a four-grid arrangement, and the three-dimensional shape of the three-dimensional sign object is a cube, the four identical third three-dimensional objects may be randomly arranged in the four-grid position to form a first component, and the same six first components may be used as the six faces of the cube, or the same six first components may be fixed to the six faces of the cube by pasting or other methods. After that, the cube may be used as a three-dimensional sign object.

[0131] In another example, at least two of the plurality of third three-dimensional objects are different objects. Optionally, the at least two third three-dimensional objects can form different objects by changing one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing. Among them, at least two of the multiple third three-dimensional objects have different three-dimensional shapes (i.e., the overall shapes of the third three-dimensional objects) (for example, there are three third three-dimensional objects, one of which is a sphere, another of which is a cube, and the remaining one is also a cube), and / or, at least two of the multiple third three-dimensional objects have different front surface shapes (for example, there are three third three-dimensional objects, one of which is a circle, another of which is a square, and the remaining one is also a square), and / or, at least two of the multiple third three-dimensional objects have different three-dimensional sizes (i.e., the overall sizes of the third three-dimensional objects) (for example, there are three third three-dimensional objects, one of which has a three-dimensional size of 15, another of which has a three-dimensional size of 10, and the remaining one is also 10), and / or, one or more of the multiple third three-dimensional objects have internal hollowing (for example, a surface (such as the front surface) of a third three-dimensional object has a groove or a hole). Exemplarily, the shape of the front surface of the third solid object may include, but is not limited to, a circle or a square (such as a square, a rectangle or other squares), etc.

[0132] For example, taking four third three-dimensional objects (such as three-dimensional object c1, three-dimensional object c2, three-dimensional object c3 and three-dimensional object c4), and the three-dimensional sizes and three-dimensional shapes of the four third three-dimensional objects are the same (for example, the three-dimensional shapes of the four third three-dimensional objects are all cubes, and the sizes of the four third three-dimensional objects are the same), and the three-dimensional shape of the three-dimensional sign object is a cube as an example, the front surface shapes of the three-dimensional objects c1, the front surface shapes of the three-dimensional objects c2 and the front surface shapes of the three-dimensional objects c4 are the same, all of which are squares, but there is a circular hole in the middle part of the front surface of the three-dimensional object c1, there is a square hole in the middle part of the front surface of the three-dimensional object c2 and the middle part of the front surface of the three-dimensional object c4, and the front surface shape of the three-dimensional object c3 is also a square, but there is neither a circular hole nor a square hole in the middle part of the front surface of the three-dimensional object c3. Afterwards, the four third three-dimensional objects can be arranged and combined according to a preset arrangement, for example, the four third three-dimensional objects can be arranged and combined according to a four-square arrangement, so that the four third three-dimensional objects can be arranged and combined in various forms at the four-square positions, thereby forming first components of various structural forms. Each structural form of the first component corresponds to a unique ID.

[0133] For example, a possible arrangement combination is: the three-dimensional object c1 is arranged in the first grid position of the first row of the four-grid, the three-dimensional object c2 is arranged in the second grid position of the first row of the four-grid, the three-dimensional object c3 is arranged in the first grid position of the second row of the four-grid, and the three-dimensional object c4 is arranged in the second grid position of the second row of the four-grid, so as to form a first component. For example, six of the first components are pasted on the six faces of the cube respectively, so as to form the following Figure 3i The three-dimensional sign object shown. Among them, the four third three-dimensional objects are arranged and combined to form Figure 3i The structure of the first component shown is unique, so that Figure 3i The first component shown corresponds to a unique ID. Exemplarily, taking the above-mentioned four three-dimensional objects arranged and transformed to form six different first components as an example, in one example, the six different first components can be directly used as the six faces of a cube, or the six different first components are pasted on the six faces of the cube respectively. In another example, when each face of the cube also includes a base plate, the six different first components are respectively combined with the base plate and used to be directly used as the six faces of the cube, or to be pasted on the six faces of the cube respectively.

[0134] As another example, taking four third three-dimensional objects (such as three-dimensional object c1, three-dimensional object c2, three-dimensional object c3 and three-dimensional object c4), and the three-dimensional sizes and shapes of the four third three-dimensional objects are the same (for example, the three-dimensional shapes of the four third three-dimensional objects are all rectangular prisms, and the sizes of the four third three-dimensional objects are the same), and the three-dimensional shape of the three-dimensional sign object is a cube, the front surface shapes of the three-dimensional objects c1, the front surface shapes of the three-dimensional objects c2, the front surface shapes of the three-dimensional objects c3 and the three-dimensional objects c4 are the same, all of which are rectangles, but there is a groove in the middle part of the front surface of the three-dimensional object c1, there is a circular hole in the middle part of the front surface of the three-dimensional object c2, there is a square hole in the middle part of the front surface of the three-dimensional object c3, and there is neither a groove nor a circular hole in the middle part of the front surface of the three-dimensional object c4. Afterwards, the four third three-dimensional objects can be arranged and combined according to a preset arrangement, for example, the four third three-dimensional objects can be arranged and combined according to a four-square arrangement, so that the four third three-dimensional objects can be arranged and combined in various forms at the four-square positions, thereby forming first components of various structural forms. Each internal component of each structural form corresponds to a unique ID.

[0135] For example, a possible arrangement combination is: the three-dimensional object c2 is arranged in the second grid position of the first row of the four-grid, the three-dimensional object c1 is arranged in the first grid position of the first row of the four-grid, the three-dimensional object c3 is arranged in the second grid position of the second row of the four-grid, and the three-dimensional object c4 is arranged in the first grid position of the second row of the four-grid, forming a new first component. Among them, the structure of the new first component formed after the arrangement and combination of the four third three-dimensional objects is also unique, so that the new first component also corresponds to a unique ID. In one example, the new first component can be directly used as a face included in the cube, or the new first component is pasted on a face included in the cube. In another example, 6 identical new first components can be directly used as the 6 faces of the cube, or 6 identical new first components are pasted on the 6 faces of the cube respectively. In another example, the above-mentioned four third three-dimensional objects can be arranged and transformed to form six different first components, and the six different first components can be directly used as six faces of the cube, or the six different first components can be pasted on the six faces of the cube respectively.

[0136] It should be noted that the above content is only an illustrative introduction to several structural forms of the first components. The above-mentioned multiple third three-dimensional objects (for example, 4 three-dimensional objects) change according to one or more of the three-dimensional shape, front surface shape, three-dimensional size or internal structure of the object, and the multiple changed third three-dimensional objects are arranged and combined according to a preset arrangement method, which can form first components with various structural forms, which are not listed one by one in the embodiments of the present application.

[0137] In another possible implementation, the sign object with a three-dimensional structure may also be a specialized sign object. For example, a combination of a corporate logo and a special three-dimensional pattern may be used as a specialized sign object, or an object of a specific shape may be added to the corporate logo to form a three-dimensional object, which is used as a specialized sign object.

[0138] Figure 4 A schematic diagram of a marking object application provided by an embodiment of the present application is exemplarily shown. Figure 4 In the example of the application of the marking object on the coal mining machine in the coal mining scene, since the body of the coal mining machine is relatively long (for example, 20 meters), it is impossible to use a sensor (such as a camera) to fully capture it at one time, so it is necessary to deploy multiple marking objects (such as 3D markers) on the coal mining machine and deploy multiple cameras (such as camera 0, camera 1, camera 2, camera 3, camera 4, camera 5 and camera 6) on the coal mining machine, so that the same camera can capture 2 or more 3D markers at a time. In order to more effectively identify the multiple 3D markers, so as to improve the accuracy of camera calibration or the accuracy of coal mining machine posture information calculation, multiple 3D markers with different IDs can be deployed on the coal mining machine, for example, 6 3D markers with different IDs are deployed on the coal mining machine, namely 3D marker0, 3D marker1, 3D marker2, 3D marker3, 3D marker4 and 3D marker5. The six 3D markers with different IDs may be formed by any combination of a plurality of internal components formed after the arrangement of the four first three-dimensional objects is transformed and a plurality of external components formed after the arrangement of the second three-dimensional objects is transformed.

[0139] For example, the above six different 3D markers can be installed on the coal mining machine in a linear arrangement, or they can be installed on the coal mining machine in other installation methods (such as an installation method arranged at any position) according to the actual application scenario. It is only necessary to measure the spatial relative position relationship of the above six different 3D markers in advance. Figure 4The 7 cameras shown in the figure can be installed in the space area where the coal mining machine is located in a linear arrangement, or can be installed in the space area where the coal mining machine is located in other installation methods (such as installation methods arranged at any position). It is only necessary to measure the spatial relative position relationship of the above 7 cameras in advance.

[0140] Step 202: When it is determined that the first image includes the first target, the image processing device performs key point detection on the first image to determine a plurality of first key points of the first target.

[0141] Optionally, after the image processing device acquires the first image, target detection can be performed on the first image. Exemplarily, the image processing device can perform target detection on the first image based on a deep learning target detection algorithm (such as a single-stage target detection algorithm such as YOLOV5 (You only look once version 5, YOLO series target detection) target detection algorithm, a single-stage target detection algorithm such as SSD (Single shot multibox detector, single-stage multi-frame detection) or a two-stage target detection algorithm such as Faster R-CNN (faster region convolutional neural networks, faster regional convolutional neural networks)) to identify the target included in the first image, and can identify the ID (or code) corresponding to the target. For example, taking the YOLOV5 target detection algorithm as an example, the image processing device uses the YOLOV5 target detection algorithm to perform target detection on the first image, and can identify the target included in the first image (such as the first target), and can identify the bounding box of the target and the code corresponding to the target (such as the first code corresponding to the first target). In this way, the image processing device uses a deep learning target detection algorithm to predict the shape, posture, texture and other attributes of the 3D landmark object from the 2D image, which helps to achieve fast and accurate target identification, and can improve the robustness of target recognition. At the same time, it can effectively deal with problems such as coal dust coverage, dust coverage or dust coverage.

[0142] In a possible implementation, when determining that the target included in the first image is the first target, the image processing device can directly perform key point detection on the first image to obtain multiple first key points of the first target. Optionally, when performing key point detection on the first image, the image processing device can also identify the categories corresponding to the multiple three-dimensional objects included in the first target.

[0143] Exemplarily, taking the example that the first target includes three three-dimensional objects (such as three-dimensional object 1, three-dimensional object 2 and three-dimensional object 3), the image processing device can use a key point detection algorithm (such as YOLO-Pose, etc.) to perform key point detection on the first image to obtain the key points contained in the first target (such as multiple key points of three-dimensional object 1, multiple key points of three-dimensional object 2 and multiple key points of three-dimensional object 3), and can also identify the categories corresponding to the three three-dimensional objects included in the first target (such as category 1 corresponding to three-dimensional object 1, category 2 corresponding to three-dimensional object 2 and category 3 corresponding to three-dimensional object 3).

[0144] In another possible implementation, in order to ensure that key points can be identified more quickly and accurately, and the robustness of key point identification is high, when it is determined that the target included in the first image is the first target, the image processing device can crop the first image according to the bounding box of the first target to obtain a target image containing the first target. Afterwards, the image processing device can perform key point detection on the target image to obtain multiple first key points of the first target. Optionally, when performing key point detection on the target image, the image processing device can also identify the categories corresponding to each of the multiple three-dimensional objects included in the first target.

[0145] Exemplarily, continuing with the example of the first target including three three-dimensional objects (such as three-dimensional object 1, three-dimensional object 2 and three-dimensional object 3), the image processing device can use a key point detection algorithm (such as YOLO-Pose, etc.) to perform key point detection on the target image to obtain the key points contained in the first target (such as multiple key points of three-dimensional object 1, multiple key points of three-dimensional object 2 and multiple key points of three-dimensional object 3), and can also identify the categories corresponding to the three three-dimensional objects included in the first target (such as category 1 corresponding to three-dimensional object 1, category 2 corresponding to three-dimensional object 2 and category 3 corresponding to three-dimensional object 3).

[0146] Exemplarily, the following takes the image to be identified as the above-mentioned target image and the key point detection algorithm as the YOLO-Pose algorithm as an example, and introduces the implementation process of the image processing device identifying key points through the following possible implementation methods.

[0147] Implementation method 1: The YOLO-Pose algorithm used by the image processing device embeds a morphological constraint algorithm, which can effectively overcome the key point distortion caused by different posture perspectives, and can effectively improve the key point recognition accuracy and robustness, and can effectively overcome the detection drift caused by dust occlusion or dust occlusion or coal dust occlusion or blurred images. Exemplarily, the morphological constraint algorithm can be embedded in the YOLO-Pose algorithm in the following way: the loss function corresponding to the morphological constraint algorithm is integrated into the loss function of the YOLO-Pose algorithm. For example, a function formula that can be used to express the loss capacity of the morphological constraint algorithm is added to the loss function of the YOLO-Pose algorithm to obtain a fusion loss function. In this way, after the training of the YOLO-Pose algorithm is completed according to the fusion loss function, the trained YOLO-Pose algorithm also has the function of the morphological constraint algorithm.

[0148] Among them, the morphological constraint algorithm refers to using more than four key points to solve the minimum energy value of the homography transformation between them and the prior key point arrangement. kpts represents the predicted value of the key points detected in the image, Taking the true value of the key point as an example, the loss capacity of the morphological constraint algorithm is measured by using the L2 norm. Among them, the loss capacity of the morphological constraint algorithm satisfies the following form:

[0149]

[0150] Among them, L H Used to indicate the loss capacity of the morphological constraint algorithm.

[0151] In order to further effectively improve the detection accuracy and posture robustness of key points, the embodiment of the present application regresses the key points included in the first target, takes the loss capacity of the morphological constraint algorithm as part of the loss function, and adds it to the loss function of the YOLO-Pose algorithm to obtain a key point regression loss function based on morphological constraints (i.e., a fusion loss function). Among them, the key point regression loss function based on morphological constraints satisfies the following form:

[0152]

[0153] Among them, L kpts It is used to represent the key point regression loss function based on morphological constraints, L P Used to represent the loss function of the YOLO-Pose algorithm, d n is the Euclidean distance between the true value and the predicted value of the nth key point, k n is the weight of the nth key point, s is the object scale factor (i.e., the scale factor of the landmark object), δ(vn >0) is the visibility flag, N kpts Used to indicate the total number of key points detected in an image (such as the total number of key points of the first target detected).

[0154] Exemplarily, when the image processing device uses the above-mentioned YOLO-Pose algorithm to identify the key points of the target image, it first detects the key points of the target image, obtains multiple first candidate key points of the first target, and verifies the multiple first candidate key points, so that the recognition accuracy of the key points can be effectively improved, and the correction of abnormal key points and the supplement of missing key points can be realized. Exemplarily, the image processing device detects the key points of the target image, and the position coordinates of multiple first candidate key points of the first target can be obtained, such as the position coordinates of multiple first candidate key points in the sensor coordinate system (such as [x, y]), which can also be understood as the position coordinates of multiple first candidate key points in the target image. In one example, when there are one or more abnormal key points in the multiple first candidate key points, the image processing device can remove the one or more abnormal key points, and the remaining multiple first candidate key points can be used as multiple first key points, so that the abnormal key points can be effectively excluded, and the calculation accuracy of the internal and external parameter information of the sensor or the positioning accuracy of the object to be photographed can be improved by removing the abnormal key points. Optionally, when the number of the remaining multiple first candidate key points is less than or equal to the quantity threshold, the image processing device can supplement at least one missing key point among the multiple first candidate key points to obtain multiple candidate key points, and the multiple candidate key points can be used as multiple first key points. In another example, when the number of the multiple first candidate key points is less than or equal to the quantity threshold, the image processing device can supplement at least one missing key point among the multiple first candidate key points to obtain multiple second candidate key points, and the multiple second candidate key points can be used as multiple first key points. In this way, by supplementing the missing key points, the calculation accuracy of the sensor's internal and external parameter information or the positioning accuracy of the object to be photographed can be effectively improved. Optionally, when there are one or more abnormal key points among the multiple second candidate key points, the image processing device can eliminate the one or more abnormal key points and use the remaining multiple second candidate key points as multiple first key points.

[0155] Implementation method 2: The YOLO-Pose algorithm used by the image processing device does not embed the morphological constraint algorithm, and the morphological constraint algorithm is used as a processing flow after the YOLO-Pose algorithm.

[0156] Among them, the image processing device first uses the YOLO-Pose algorithm to detect key points of the target image to obtain multiple first candidate key points of the first target. Exemplarily, the image processing device uses the YOLO-Pose algorithm to detect key points of the target image, and can obtain the position coordinates of multiple first candidate key points of the first target, such as the position coordinates of multiple first candidate key points in the sensor coordinate system (such as [x, y]), which can also be understood as the position coordinates of multiple first candidate key points in the target image. Afterwards, the image processing device uses a morphological constraint algorithm to verify the multiple first candidate key points, so that the correction of abnormal key points and the supplement of missing key points can be achieved. In one example, when there are one or more abnormal key points in the multiple first candidate key points, the image processing device can eliminate the one or more abnormal key points, and the remaining multiple first candidate key points can be used as multiple first key points, so that the abnormal key points can be effectively excluded, and the calculation accuracy of the internal and external parameter information of the sensor or the positioning accuracy of the object to be photographed can be improved by eliminating the abnormal key points. Optionally, when the number of the remaining multiple first candidate key points is less than or equal to the quantity threshold, the image processing device can supplement at least one missing key point among the multiple first candidate key points to obtain multiple candidate key points, and the multiple candidate key points can be used as multiple first key points. In another example, when the number of the multiple first candidate key points is less than or equal to the quantity threshold, the image processing device can supplement at least one missing key point among the multiple first candidate key points to obtain multiple second candidate key points, and the multiple second candidate key points can be used as multiple first key points. In this way, by supplementing the missing key points, the calculation accuracy of the sensor's internal and external parameter information or the positioning accuracy of the object to be photographed can be effectively improved. Optionally, when there are one or more abnormal key points among the multiple second candidate key points, the image processing device can eliminate the one or more abnormal key points and use the remaining multiple second candidate key points as multiple first key points.

[0157] Optionally, before verifying the multiple first candidate key points, the image processing device also needs to verify the code corresponding to the first target identified by the target detection algorithm to determine whether the first target identified by the target detection algorithm is accurate, thereby achieving double verification of the first target, and when the first target is identified incorrectly, there is no need to perform subsequent processing procedures, so as to avoid the waste of subsequent computing resources caused by the incorrect identification of the first target. When the image processing device verifies the code corresponding to the first target identified by the target detection algorithm, it first selects the center position point or corner point of the multiple three-dimensional objects included in the first target from the multiple first candidate key points. For example, the image processing device selects which candidate key points belong to the center position point of the three-dimensional object from the multiple first candidate key points, or selects which candidate key points belong to the corner point of the three-dimensional object from the multiple first candidate key points. Among them, the corner point is used to indicate the intersection between multiple contour lines in the three-dimensional object. For example, if the three-dimensional object is a cube object, the corner point can refer to each vertex of the cube object. Afterwards, the image processing device can determine the arrangement and combination mode of the multiple three-dimensional objects (i.e., the structural form of the first target) according to the center position point or corner point of the multiple three-dimensional objects included in the first target. Among them, each structural form of the marking object corresponds to a unique code. Then, the image processing device can determine the second code corresponding to the first target according to the arrangement combination of multiple three-dimensional objects. For example, taking the center position point of the three-dimensional object as an example, assuming that the first target includes three three-dimensional objects (such as three-dimensional object 1, three-dimensional object 2 and three-dimensional object 3), the image processing device can determine the arrangement position of three-dimensional object 1, the arrangement position of three-dimensional object 2 and the arrangement position of three-dimensional object 3 according to the center position point of three-dimensional object 1, the center position point of three-dimensional object 2 and the center position point of three-dimensional object 3, so as to determine the structural form of the first target, and thus determine the second code corresponding to the first target.

[0158] When the image processing device determines that the second code corresponding to the first target is the same as the first code corresponding to the first target, the verification process and subsequent processing process for multiple first candidate key points can be executed. When the image processing device determines that the second code corresponding to the first target is different from the first code corresponding to the first target, there is no need to execute the subsequent processing process for the first target (such as the process of the image processing device verifying multiple first candidate key points, or the process of the image processing device calculating the internal and external parameter information of the sensor or the posture information of the object to be photographed based on the multiple first key points).

[0159] Figure 5 A schematic diagram of key point regression provided by an embodiment of the present application is exemplified. Figure 5In the example, the first target is a certain 3D marker, and the image processing device uses the YOLO-Pose algorithm embedded in the morphological constraint algorithm. When the image processing device uses the above YOLO-Pose algorithm to identify key points of the target image containing the 3D marker, the target image is firstly detected to obtain 6 candidate key points of the 3D marker, namely, candidate key point 0, candidate key point 1, candidate key point 2, candidate key point 3, candidate key point 4 and candidate key point 5, and the 6 candidate key points are verified by the morphological constraint algorithm. When there are no abnormal key points among the 6 candidate key points and the number of the 6 candidate key points is greater than the number threshold (such as 3), the image processing device uses the 6 candidate key points as target key points for calculating the internal and external reference information of the sensor (or the position and posture information of the object to be photographed).

[0160] Optionally, after executing step 202, the image processing device may further execute step 203. By executing step 203, the internal and external parameter information of the first sensor may be calculated, thereby achieving accurate calibration of the first sensor, or the posture information of the object to be photographed may be calculated by executing step 203, thereby achieving accurate positioning of the object to be photographed.

[0161] Step 203: The image processing device determines the internal and external parameter information of the first sensor or the position and posture information of the object to be photographed according to the multiple first key points.

[0162] Optionally, after the image processing device obtains multiple first key points, that is, the image processing device obtains the position coordinates of the multiple first key points in the first image (or in the target image), it can calculate the internal and external parameter information of the first sensor according to the position coordinates of the multiple first key points in the first image (or in the target image) and the known spatial position coordinates of the multiple first key points in the world coordinate system, or it can also calculate the posture information of the object to be photographed.

[0163] Exemplarily, taking the first sensor as a camera, when multiple first key points are used to calibrate the camera, the image processing device can use the PnP algorithm to perform PnP calculations on the coordinate positions of the multiple first key points in the camera coordinate system and the spatial coordinate positions of the multiple first key points in the world coordinate system, and calculate the internal and external parameter information of the camera. When multiple first key points are used to locate the object to be photographed, the image processing device can use the PnP algorithm to perform PnP calculations on the coordinate positions of the multiple first key points in the camera coordinate system and the spatial coordinate positions of the multiple first key points in the world coordinate system, and calculate the pose information of the object to be photographed.

[0164] It should be noted that in the description of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. "At least one of the following (individuals)" or its similar expression refers to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC or ABC. And, unless otherwise specified, the ordinal numbers such as "first", "second", and "third" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. In addition, the terms "including", "comprising", "having" and their variations appearing in the present application all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0165] In addition, it should be noted that each step involved in the above embodiments can be performed by a corresponding device, or by a chip, processor, or chip system in the device, and the embodiments of the present application do not limit them. The above embodiments are only described by taking the corresponding device as an example.

[0166] It should be noted that in the above embodiments, some steps can be selected for implementation, and the order of the steps in the diagram can be adjusted for implementation, and this application does not limit this. It should be understood that executing some steps in the diagram, adjusting the order of the steps, or combining them for specific implementation all fall within the scope of protection of this application.

[0167] It is understandable that, in order to implement the functions in the above embodiments, the various devices involved in the above embodiments include hardware structures and / or software modules corresponding to the execution of the various functions. It should be easily appreciated by those skilled in the art that, in combination with the units and method steps of the various examples described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0168] It should be noted that the "steps" in the embodiments of the present application are only for illustration, and are a method of expression used to better understand the embodiments, and do not constitute a substantial limitation on the execution of the scheme of the present application. For example, the "steps" can also be understood as "features". In addition, the steps do not constitute any limitation on the execution order of the scheme of the present application. Any changes in the order of steps, step merging, or step splitting made on this basis that do not affect the implementation of the overall scheme, and the resulting new technical solutions are also within the scope of the disclosure of the present application.

[0169] Based on the same concept, the present application also provides a possible image processing device, which is suitable for Figure 1a to Figure 1d In the application scenario shown. Exemplarily, the image processing device may be a computing device having the functions required to implement the method, or may be a functional element (such as a plug-in, component or chip, etc.) having the functions required to implement the method, or may be any other device (such as a robot device, a cloud server or a vehicle-mounted device, etc.) having the functions required to implement the method, or may be a functional element set in a computing device or any other device. Optionally, the image processing device is used to implement the image processing method provided in the above embodiment, or the module (such as a chip) of the image processing device is used to implement the image processing method provided in the above embodiment, so the beneficial effects of the above embodiment can also be achieved. Exemplarily, taking the image processing device as a chip set in a computing device as an example, when the image processing device is a chip, the image processing device includes a transceiver and a processor, but does not include a memory. Among them, the transceiver exists as an input and output interface, and the input and output interface is used for the chip to realize the transceiver of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can realize the reception of the computing device, and the output interface can be used to realize the sending of the computing device. The processor is used to read and execute corresponding computer programs or instructions so that the corresponding functions of the image processing device are realized. Optionally, when the chip implements the corresponding functions of the image processing device in the above embodiment, the input and output interface can implement the transceiver operation performed by the image processing device in the above embodiment; the processor can implement other operations except the transceiver operation performed by the image processing device in the above embodiment. For specific related descriptions, please refer to the related descriptions in the above embodiment, which will not be described in detail here.

[0170] See also Figure 6 The image processing device 600 includes an acquisition module 601 (or may be called a transceiver module, used for sending and receiving data) and a processing module 602.

[0171] Optionally, the acquisition module 601 may include a receiving module and / or a sending module. The receiving module may be used for the image processing device 600 to receive a signal (information or data, etc.); the sending module may be used for the image processing device 600 to send a signal (information or data, etc.). The sending module may send a signal (information or data, etc.) under the control of the processing module 602, and the receiving module may receive a signal (information or data, etc.) under the control of the processing module 602.

[0172] When the image processing device 600 is used to implement Figure 2 The function of the image processing device in the method embodiment shown is as follows: the acquisition module 601 is used to acquire the first image. The first image is obtained by photographing the object to be photographed by the first sensor. The processing module 602 is used to perform key point detection on the first image when it is determined that the first target is included in the first image, and determine multiple first key points of the first target. The first target is a landmark object with a three-dimensional structure, the first target is located on the object to be photographed, the first key point is used to characterize the structural features of the first target, and the multiple first key points are used to determine the internal and external parameter information of the first sensor, or to determine the posture information of the object to be photographed.

[0173] For a more detailed description of the acquisition module 601 and the processing module 602, reference may be made to the relevant description of the image processing device in the above method embodiment, which will not be repeated here.

[0174] It should be understood that the acquisition module 601 in the embodiment of the present application can be implemented by a transceiver or a transceiver-related circuit component, and the processing module 602 can be implemented by a processor or a processor-related circuit component.

[0175] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated in a processing unit, or may exist physically separately, or two or more units may be integrated in one unit. For example, the acquisition module and the processing module may be integrated in one module, or the acquisition module and the processing module may be the same module. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, or a server, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0177] Based on the same concept, the present application also provides a possible image processing device, which is suitable for Figure 1a to Figure 1d The image processing device is used to implement the technical solutions involved in the image processing device in the above embodiments, so it can also achieve the beneficial effects of the image processing device in the above method embodiments. Figure 7 , the image processing device 700 includes: a transceiver 701 and a processor 702. Optionally, the image processing device 700 also includes a memory 703. The transceiver 701, the processor 702 and the memory 703 are interconnected. When the image processing device 700 is used to implement the technical solution involved in the image processing device provided in the above embodiment, the transceiver 701 can be used to implement the function of the above acquisition module 601, and the processor 702 is used to implement the function of the above processing module 602.

[0178] Optionally, the transceiver 701, the processor 702, and the memory 703 are interconnected via a bus 704. The bus 704 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0179] The transceiver 701 is used to receive and send data. For example, when the image processing device is a computing device (such as a computer) having the functions required to implement the method, the transceiver 701 can be used to communicate with a sensor (such as a camera or a laser radar) or other devices (such as a robot device or a cloud server). In one example, the transceiver can be a transceiver device with integrated data transceiver function. In another example, the transceiver can also be composed of a transmitter and a receiver, wherein the transmitter is used to send data and the receiver is used to receive data.

[0180] Optionally, the transceiver 701 may include a transmitter and / or a receiver. The transmitter is used to send a signal, message, information, or data, etc. The receiver is used to receive a signal, message, information, or data, etc. Exemplarily, the transmitter sends a signal, message, information, or data, etc. under the control of the processor 702. The receiver receives a signal, message, information, or data, etc. under the control of the processor 702.

[0181] The functions of the processor 702 can refer to the description of the corresponding functions involved in the image processing device in the above embodiment, and will not be repeated here. Among them, the processor 702 can be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP, etc. The processor 702 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. When the processor 702 implements the above-mentioned functions, it can be implemented by hardware, and of course, it can also be implemented by executing the corresponding software through hardware.

[0182] The memory 703 is used to store program instructions, etc. Specifically, the program instructions may include program codes, and the program codes include computer operation instructions. The memory 703 may include random access memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. The processor 702 executes the program instructions stored in the memory 703 to implement the above functions, thereby implementing the method steps required to be executed by the image processing device in the above embodiment.

[0183] Based on the same concept, the embodiment of the present application also provides a possible image processing system, which includes a marker object with a three-dimensional structure, a sensor, and an image processing device, wherein the image processing device can be used to implement the technical solution involved in the image processing device in the above embodiment, the marker object can be used to realize the calibration of the sensor or the positioning of the object to be photographed (the marker object is deployed on the object to be photographed), and the sensor can be used to capture the image information of the marker object or to capture the environmental information of the area where the marker object is located. Exemplarily, the number of the marker objects included in the image processing system can be one or more, the number of the sensors can be one or more, and the number of the image processing devices can be one or more, and the embodiment of the present application does not limit this.

[0184] Based on the same concept, an embodiment of the present application further provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer executes the method provided in the above embodiment.

[0185] Based on the same concept, an embodiment of the present application also provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a computer, the computer executes the method provided in the above embodiment.

[0186] The storage medium may be any available medium that can be accessed by a computer. For example, but not limited to, a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0187] Based on the same concept, an embodiment of the present application further provides a chip, which is coupled to a memory, and is used to read a computer program stored in the memory to implement the method provided in the above embodiment.

[0188] Based on the same concept, the embodiment of the present application also provides a chip system, which includes a processor for supporting a computer device to implement the functions involved in the image processing device in the above embodiment. In a possible design, the chip system also includes a memory, which is used to store the necessary programs and data for the computer device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0189] In the method provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0190] The steps of the method described in the embodiments of the present application can be directly embedded in the hardware, the software unit executed by the processor, or the combination of the two. The software unit can be stored in RAM, ROM, EEPROM, register, hard disk, removable disk, CD-ROM or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC.

[0191] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0193] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, where the first image is obtained by photographing the object to be photographed by a first sensor; When it is determined that the first image contains a first target, performing key point detection on the first image to determine a plurality of first key points of the first target, wherein the first target is a landmark object with a three-dimensional structure, the first target is located on the object to be photographed, and the first key points are used to characterize structural features of the first target; The multiple first key points are used to determine the internal and external parameter information of the first sensor, or to determine the position and posture information of the object to be photographed.

2. The method according to claim 1, characterized in that The first target includes an internal component, and the internal component is composed of a plurality of first three-dimensional objects arranged and combined in a preset arrangement manner, and there are first gaps between the plurality of first three-dimensional objects.

3. The method according to claim 2, characterized in that All structural properties of the plurality of first three-dimensional objects are the same, or at least two of the plurality of first three-dimensional objects are different in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

4. The method according to claim 2 or 3, characterized in that The first target also includes an external component, which is composed of a plurality of second three-dimensional objects arranged and combined in a preset arrangement manner. The external component is used to surround the internal component, and a second gap is provided between the external component and the internal component.

5. The method according to claim 4, characterized in that There are third gaps between the plurality of second solid objects, and through holes are arranged around the outer component.

6. The method according to claim 4 or 5, characterized in that All structural properties of the plurality of second three-dimensional objects are the same, or at least two of the plurality of second three-dimensional objects are different in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

7. The method according to claim 1, characterized in that The first target is also a three-dimensional sign object with a curved surface or a three-dimensional sign object with multiple planes, wherein each surface included in the first target includes a first component, and the first component is composed of a plurality of third three-dimensional objects arranged and combined in a preset arrangement, and there are fourth gaps between the plurality of third three-dimensional objects.

8. The method according to claim 7, characterized in that All structural properties of the plurality of third three-dimensional objects are the same, or at least two third three-dimensional objects among the plurality of third three-dimensional objects are different in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

9. The method according to any one of claims 2 to 8, characterized in that: The first target further includes a base plate, the base plate being used to carry one or more of the following: an internal component, an external component, or a first component.

10. The method according to any one of claims 1 to 9, characterized in that: Performing key point detection on the first image to determine a plurality of first key points of the first object includes: Performing key point detection on the first image, determining a plurality of first candidate key points of the first object, and verifying the plurality of first candidate key points; When there are abnormal key points among the multiple first candidate key points, the abnormal key points are eliminated, and the remaining multiple first candidate key points are used as the multiple first key points; or When the number of the multiple first candidate key points is less than or equal to the number threshold, at least one missing key point is supplemented in the multiple first candidate key points to obtain multiple second candidate key points, and the multiple second candidate key points are used as the multiple first key points.

11. The method according to claim 10, characterized in that Before verifying the plurality of first candidate key points, the method further includes: Determine that a first code corresponding to the first target is the same as a second code corresponding to the first target; Among them, the second code is determined according to the arrangement and combination of the three-dimensional objects included in the first target, the arrangement and combination of the three-dimensional objects included in the first target is determined according to the center position point or corner point of the three-dimensional objects included in the first target, the center position point or corner point of the three-dimensional objects included in the first target is selected from the multiple first candidate key points, the first code is obtained by performing target detection on the first image, the first code or the second code is used to identify the first target, and the corner point is used to indicate the intersection between multiple contour lines in the three-dimensional object.

12. The method according to claim 10 or 11, characterized in that Performing key point detection on the first image to determine a plurality of first candidate key points of the first object includes: Cropping the first image according to a bounding box of a first target to obtain a target image including the first target, where the bounding box of the first target is obtained by performing target detection on the first image; Key point detection is performed on the target image to determine a plurality of first candidate key points of the first target.

13. An image processing device, characterized in that: include: An acquisition module, used to acquire a first image, where the first image is obtained by photographing the object to be photographed by a first sensor; A processing module is used to perform key point detection on the first image to determine multiple first key points of the first target when it is determined that the first image contains a first target, wherein the first target is a landmark object with a three-dimensional structure, the first target is located on the object to be photographed, and the first key points are used to characterize the structural features of the first target; wherein the multiple first key points are used to determine the internal and external parameter information of the first sensor, or to determine the posture information of the object to be photographed.

14. The device according to claim 13, characterized in that When the processing module performs key point detection on the first image to determine a plurality of first key points of the first target, the processing module is specifically used to: Performing key point detection on the first image, determining a plurality of first candidate key points of the first object, and verifying the plurality of first candidate key points; When there are abnormal key points among the multiple first candidate key points, the abnormal key points are eliminated, and the remaining multiple first candidate key points are used as the multiple first key points; or When the number of the multiple first candidate key points is less than or equal to the number threshold, at least one missing key point is supplemented in the multiple first candidate key points to obtain multiple second candidate key points, and the multiple second candidate key points are used as the multiple first key points.

15. The device according to claim 14, characterized in that Before verifying the plurality of first candidate key points, the processing module is further configured to: Determine that a first code corresponding to the first target is the same as a second code corresponding to the first target; Among them, the second code is determined according to the arrangement and combination of the three-dimensional objects included in the first target, the arrangement and combination of the three-dimensional objects included in the first target is determined according to the center position point or corner point of the three-dimensional objects included in the first target, the center position point or corner point of the three-dimensional objects included in the first target is selected from the multiple first candidate key points, the first code is obtained by performing target detection on the first image, the first code or the second code is used to identify the first target, and the corner point is used to indicate the intersection between multiple contour lines in the three-dimensional object.

16. The device according to claim 14 or 15, characterized in that When the processing module performs key point detection on the first image to determine a plurality of first candidate key points of the first target, the processing module is specifically used to: Cropping the first image according to a bounding box of a first target to obtain a target image including the first target, where the bounding box of the first target is obtained by performing target detection on the first image; Key point detection is performed on the target image to determine a plurality of first candidate key points of the first target.

17. An image processing device, characterized in that: including a transceiver, a memory and a processor; The transceiver is used to receive and send data; The memory is used to store computer program instructions and data; The processor is used to execute and call the computer program instructions and data in the memory, so that the image processing device performs the method as claimed in any one of claims 1-12.

18. A marking object having a three-dimensional structure, characterized in that: The marking object includes an internal component, and the internal component is composed of a plurality of first three-dimensional objects arranged and combined in a preset arrangement manner, and there are first gaps between the plurality of first three-dimensional objects.

19. The marking object according to claim 18, characterized in that: All structural properties of the plurality of first three-dimensional objects are the same, or at least two of the plurality of first three-dimensional objects are different in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

20. The marking object according to claim 18 or 19, characterized in that: The marking object also includes an external component, which is composed of a plurality of second three-dimensional objects arranged and combined in a preset arrangement. The external component is used to surround the internal component, and a second gap is provided between the external component and the internal component.

21. The marking object according to claim 20, characterized in that: There are third gaps between the plurality of second solid objects, and through holes are arranged around the outer component.

22. The marking object according to claim 20 or 21, characterized in that: All structural properties of the plurality of second three-dimensional objects are the same, or at least two of the plurality of second three-dimensional objects are different in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

23. A marking object having a three-dimensional structure, characterized in that: The marking object is a three-dimensional marking object with a curved surface or a three-dimensional marking object with multiple planes, wherein each surface of the three-dimensional marking object with a curved surface or the three-dimensional marking object with multiple planes includes a first component, and the first component is composed of a plurality of third three-dimensional objects arranged and combined in a preset arrangement manner, and there is a fourth gap between the plurality of third three-dimensional objects.

24. The marking object according to claim 23, characterized in that: All structural properties of the plurality of third three-dimensional objects are the same, or at least two third three-dimensional objects among the plurality of third three-dimensional objects are different in one or more of the following structural properties: three-dimensional shape, front surface shape, three-dimensional size, or whether there is internal hollowing.

25. The marking object according to any one of claims 18 to 24, characterized in that: The sign object further includes a base plate, and the base plate is used to carry one or more of the following: an internal component, an external component, or a first component.

26. An image processing system, characterized in that: It includes a sensor, a marking object with a three-dimensional structure as described in any one of claims 18 to 25, and an image processing device as described in any one of claims 13 to 17, wherein the marking object is used to realize the calibration of the sensor or the positioning of the object to be photographed, and the sensor is used to capture the image information of the marking object or to capture the environmental information of the area where the marking object is located.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction. When the computer program or instruction is executed by a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

28. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 12.

29. A chip, characterized in that: The chip is coupled to a memory, and the chip reads a computer program stored in the memory to execute the method according to any one of claims 1 to 12.

Images