Multi-target three-dimensional positioning method and device based on mobile monocular camera

Abstract

The invention provides a multi-target three-dimensional positioning method and device based on a moving monocular camera, relates to the technical field of target detection, and is applied to a moving mechanism which comprises a monocular camera, a first sliding rod, a supporting rod and a second sliding rod. The monocular camera can horizontally move relative to the first sliding rod, and the supporting rod can horizontally move relative to the second sliding rod. The method comprises the steps that a supporting rod is controlled to move to an observation point position of a second sliding rod, and a monocular camera is controlled to move from a moving starting point of a first sliding rod to a moving end point of the first sliding rod; and under the condition of traversing all the observation points of the second sliding rod, fusing the three-dimensional coordinates of the same target at the plurality of observation points to obtain a three-dimensional fused coordinate of the same target. According to the invention, a multi-virtual binocular system with a wider field of view is constructed from multiple angles and multiple observation points, and the three-dimensional coordinates of the same target at the multiple observation points are fused, so that the target positioning precision is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a multi-target three-dimensional positioning method and device based on a mobile monocular camera. Background Art

[0002] In agricultural production, accurate positioning of harvesting targets plays a vital role in achieving efficient and automated crop harvesting.

[0003] The traditional positioning method is mainly: first, through the preliminary 3D reconstruction of the working scene, a point cloud model of the scene is constructed. Then, the real-time image captured by the monocular camera is used to match the pre-established 3D point cloud model to obtain the scene depth information perceived by the current camera position. Finally, by analyzing the real-time depth map captured by the monocular camera, the target object can be accurately positioned in 3D space.

[0004] However, agricultural harvesting scenes are complex and dynamically changing. When the scene changes, the scene needs to be reconstructed when using monocular point cloud matching. The quality of scene reconstruction will have a great impact on positioning accuracy. In addition, monocular point cloud matching has large errors, resulting in insufficient target positioning accuracy. Summary of the invention

[0005] The present invention provides a multi-target three-dimensional positioning method and device based on a mobile monocular camera, which are used to solve the defect of insufficient target positioning accuracy in the prior art and improve the target positioning accuracy.

[0006] The present invention provides a multi-target three-dimensional positioning method based on a mobile monocular camera, which is applied to a mobile mechanism, wherein the mobile mechanism includes a monocular camera, a first sliding rod, a support rod and a second sliding rod; The monocular camera is arranged on the first sliding bar, and the monocular camera can move horizontally relative to the first sliding bar. The first sliding bar is connected to the second sliding bar through the supporting bar, and the supporting bar can move horizontally relative to the second sliding bar. The movable range of the second sliding bar is greater than the movable range of the first sliding bar. A plurality of observation points are arranged within the movable range of the second sliding bar. The method comprises: Controlling the support rod to move to the observation point of the second sliding rod; When the support rod is at the current observation point, control the monocular camera to move from the moving starting point of the first sliding rod to the moving end point of the first sliding rod, and acquire a first image captured by the monocular camera at the moving starting point of the first sliding rod and a second image captured by the monocular camera at the moving end point of the first sliding rod; Based on a deep neural network, perform multi-target recognition on the first image and the second image respectively, and obtain a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; Based on the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of each target, the three-dimensional coordinates of each target at the current observation point are acquired; When all observation points of the second slide bar are traversed, the three-dimensional coordinates of the same target at multiple observation points are fused to obtain the three-dimensional fused coordinates of the same target.

[0007] In some embodiments, fusing the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fused coordinates of the same target includes: Calculate the average value of the three-dimensional coordinates of the same target at multiple observation points; The average value is used as the three-dimensional fusion coordinates of the same target.

[0008] In some embodiments, before fusing the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fused coordinates of the same target, the method further includes: Calculate the spatial distance between any two targets based on the three-dimensional coordinates of each target at different observation points; Two targets whose spatial distance is less than a preset spatial distance are determined to be the same target.

[0009] In some embodiments, based on the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of each target, obtaining the three-dimensional coordinates of each target at the current observation point includes: Acquire the vertical axis coordinate of each target at the current observation point according to the focal length of the monocular camera, the moving distance between the moving starting point and the moving end point of the first sliding bar, and the horizontal coordinate difference between the first pixel coordinate and the second pixel coordinate of each target; Obtaining the horizontal axis coordinate of each target at the current observation point according to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the horizontal coordinate of each target in the first pixel coordinate; The vertical axis coordinate of each target at the current observation point is obtained according to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the vertical coordinate of each target in the first pixel coordinate.

[0010] In some embodiments, the method further comprises: In the process of controlling the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, each target is continuously tracked based on a multi-target tracking algorithm.

[0011] In some embodiments, the deep neural network is YOLOv8.

[0012] The present invention also provides a multi-target three-dimensional positioning device based on a mobile monocular camera, which is applied to a mobile mechanism, wherein the mobile mechanism includes a monocular camera, a first sliding rod, a support rod, and a second sliding rod; The monocular camera is arranged on the first sliding bar, and the monocular camera can move horizontally relative to the first sliding bar. The first sliding bar is connected to the second sliding bar through the supporting bar, and the supporting bar can move horizontally relative to the second sliding bar. A plurality of observation points are arranged on the second sliding bar. The movable range of the second sliding bar is greater than the movable range of the first sliding bar. The device comprises: A first control module, used for controlling the support rod to move to the observation point of the second sliding rod; a second control module, configured to control the monocular camera to move from a moving starting point of the first sliding bar to a moving end point of the first sliding bar when the support bar is at a current observation point, and to obtain a first image taken by the monocular camera at the moving starting point of the first sliding bar and a second image taken by the monocular camera at the moving end point of the first sliding bar; A target recognition module, used to perform multi-target recognition on the first image and the second image based on a deep neural network, and obtain a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; an acquisition module, configured to acquire the three-dimensional coordinates of each target at a current observation point based on the moving distance between the moving starting point and the moving end point of the first sliding bar and the first pixel coordinates and the second pixel coordinates of each target; The fusion module is used to fuse the three-dimensional coordinates of the same target at multiple observation points while traversing all observation points of the second sliding bar to obtain the three-dimensional fused coordinates of the same target.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multi-target three-dimensional positioning method based on a mobile monocular camera as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described multi-target three-dimensional positioning methods based on a mobile monocular camera.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for three-dimensional positioning of multiple targets based on a mobile monocular camera.

[0016] The multi-target three-dimensional positioning method and device based on a mobile monocular camera provided by the present invention constructs a multi-virtual binocular system with a wider field of view from multiple angles and multiple observation points by controlling the movement of the monocular camera on a first sliding bar and controlling the movement of a supporting rod on a second sliding bar, and fuses the three-dimensional coordinates of the same target at multiple observation points, thereby effectively improving the accuracy of target positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a structural schematic diagram of the mobile mechanism provided by the present invention; Figure 2 It is a flow chart of a multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention; Figure 3 It is a schematic diagram of the principle of virtual binocular provided by the present invention; Figure 4 It is a schematic diagram of multi-target tracking and positioning provided by the present invention; Figure 5 It is an overall workflow diagram of the multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention; Figure 6 It is a schematic diagram of positioning based on a monocular camera provided by the present invention; Figure 7 It is a structural schematic diagram of a multi-target three-dimensional positioning device based on a mobile monocular camera provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 It is a structural schematic diagram of the mobile mechanism provided by the present invention, such as Figure 1 As shown, the moving mechanism includes a monocular camera, a first sliding bar, a support bar, and a second sliding bar. The monocular camera is arranged on the first sliding bar, and the monocular camera can move horizontally relative to the first sliding bar. The first sliding bar is connected to the second sliding bar through the support bar, and the support bar can move horizontally relative to the second sliding bar. The movable range of the second sliding bar is greater than the movable range of the first sliding bar, and a plurality of observation points are arranged within the movable range of the second sliding bar.

[0021] It should be noted that the present invention does not specifically limit the shapes of the first slide bar and the second slide bar, which may be a straight line, a curve, or a broken line, or a combination of at least two of the straight line, the curve, and the broken line. The specific shapes of the first slide bar and the second slide bar can be reasonably set according to the actual agricultural harvesting scene to ensure that the shooting range of the monocular camera is as large as possible.

[0022] It should be noted that the present invention does not specifically limit the height of the support rod. The height of the support rod can be reasonably set according to the actual agricultural harvesting scene to ensure that the monocular camera can capture the harvesting target.

[0023] Applied to the above-mentioned mobile mechanism, the present invention provides a multi-target three-dimensional positioning method based on a mobile monocular camera. Figure 2 : is a flow chart of a multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention, such as Figure 2 As shown, the following steps are included: Step 210, controlling the support rod to move to the observation point of the second sliding rod.

[0024] Specifically, the moving distance and moving speed of the support rod are controllable, and the support rod is controlled to move to the observation point of the second slide bar. In order to facilitate the movement according to the sorting order of the observation points, the observation point here can be the first observation point on the second slide bar.

[0025] Step 220, when the support rod is at the current observation point, control the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, and obtain a first image taken by the monocular camera at the moving starting point of the first slide bar and a second image taken by the monocular camera at the moving end point of the first slide bar.

[0026] Specifically, at the current observation point, first, the monocular camera is controlled to capture an image at the moving starting point of the first slider, and a first image captured by the monocular camera at the moving starting point of the first slider is obtained; then, the monocular camera is controlled to move from the moving starting point of the first slider to the moving end point of the first slider and capture an image, and a second image captured by the monocular camera at the moving end point of the first slider is obtained.

[0027] Figure 3 Schematic diagram of the principle of virtual binocular provided by the present invention, such as Figure 3 As shown, the monocular camera moves from the moving start point of the first slider to the moving end point of the first slider. Since parallax can be formed between the moving start point and the moving end point of the first slider, the shooting effect of the monocular camera at the moving start point and the moving end point of the first slider is equivalent to the shooting effect of the binocular camera. Therefore, the combination of the monocular camera and the first slider can be regarded as a virtual binocular system.

[0028] On the basis of the constructed virtual binocular system, combined with the second sliding bar with a larger movable range, multiple observation points are set on the second sliding bar, and the support rod is controlled to move at the observation point to drive the virtual binocular system to move horizontally, thereby constructing a virtual binocular system for multi-point observation. By forming a virtual binocular system for multi-point observation, the field of view limitation of the monocular camera is reduced and the measurement range is expanded.

[0029] Step 230: Perform multi-target recognition on the first image and the second image based on a deep neural network to obtain the first pixel coordinates of each target in the first image and the second pixel coordinates of each target in the second image.

[0030] Specifically, after acquiring the first image, a deep neural network is used to perform multi-target recognition on the first image to obtain the first pixel coordinates of each target in the first image. After acquiring the second image, a deep neural network is used to perform multi-target recognition on the second image to obtain the first pixel coordinates of each target in the second image.

[0031] In some embodiments, the deep neural network is YOLOv8.

[0032] Step 240 , based on the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of each target, obtain the three-dimensional coordinates of each target at the current observation point.

[0033] Specifically, the three-dimensional coordinates of each target at the current observation point can be obtained by using the pixel coordinates of each target in the first image and the second image, combined with the moving distance between the moving start point and the moving end point based on the first slide bar.

[0034] Step 250, determining whether all observation points of the second slider have been traversed.

[0035] Specifically, it is determined whether all observation points of the second slide bar have been traversed. If it is determined that all observation points of the second slide bar have not been traversed, steps 210 to 250 are performed on the next observation point. If it is determined that all observation points of the second slide bar have been traversed, step 260 is performed.

[0036] Step 260, fusion of the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fusion coordinates of the same target.

[0037] Specifically, if the same target is observed at multiple observation points, the three-dimensional coordinates of the same target at the multiple observation points are fused to obtain the three-dimensional fused coordinates of the same target, so as to improve the positioning accuracy of the same target.

[0038] The multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention constructs a multi-virtual binocular system with a wider field of view from multiple angles and multiple observation points by controlling the movement of the monocular camera on a first sliding bar and controlling the movement of a supporting rod on a second sliding bar, and fuses the three-dimensional coordinates of the same target at multiple observation points, thereby effectively improving the accuracy of target positioning.

[0039] In addition, this method can not only significantly reduce the computing load, but also reduce the interference of factors such as light in complex environments, thereby achieving more accurate and reliable target positioning, greatly improving practicality and application prospects, and providing new solutions for the development of smart agriculture.

[0040] In some embodiments, the three-dimensional coordinates of the same target at multiple observation points are fused to obtain the three-dimensional fused coordinates of the same target, including: Calculate the average value of the three-dimensional coordinates of the same target at multiple observation points; The average value is used as the 3D fusion coordinate of the same target.

[0041] Specifically, for example, the three-dimensional coordinates of the target at observation point 1 are (X W1 , Y W1 , Z W1 ), the three-dimensional coordinates of the target at observation point 2 are (X W2 , Y W2 , Z W2 ), then the three-dimensional fusion target of the target is ((X W1 +X W2 ) / 2, (Y W1 +Y W2 ) / 2, (Z W1 +Z W2 ) / 2).

[0042] The multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention further helps to improve the positioning accuracy by taking the average value of the three-dimensional coordinates of the same target at multiple observation points as the three-dimensional fused coordinates of the same target.

[0043] In some embodiments, before fusing the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fused coordinates of the same target, the method further includes: Calculate the spatial distance between any two targets based on the three-dimensional coordinates of each target at different observation points; Two targets whose spatial distance is less than a preset spatial distance are determined to be the same target.

[0044] Specifically, since different observation points may observe the same target, after obtaining the three-dimensional coordinates of each target at different observation points, the spatial distance between any two targets is calculated based on the three-dimensional coordinates of the any two targets.

[0045] The spatial distance between the two targets is compared with the preset spatial distance. The preset spatial distance can be reasonably set according to the actual agricultural harvesting scene to ensure the accuracy of the judgment of the same target. If the spatial distance between the two targets is less than the preset spatial distance, it is determined that the two targets are the same target. If the spatial distance between the two targets is greater than or equal to the preset spatial distance, it is determined that the two targets are not the same target.

[0046] The multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention can simply realize the determination of the same target by determining two targets whose spatial distance is less than a preset spatial distance as the same target.

[0047] In some embodiments, based on the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of each target, the three-dimensional coordinates of each target at the current observation point are obtained, including: According to the focal length of the monocular camera, the moving distance between the moving starting point and the moving end point of the first slider, and the horizontal coordinate difference between the first pixel coordinate and the second pixel coordinate of each target, the vertical axis coordinate of each target at the current observation point is obtained; According to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the horizontal coordinate in the first pixel coordinate of each target, the horizontal axis coordinate of each target at the current observation point is obtained; The vertical coordinate of each target at the current observation point is obtained according to the focal length of the monocular camera, the vertical coordinate of each target at the current observation point, and the vertical coordinate in the first pixel coordinate of each target.

[0048] Specifically, for example, the first pixel coordinate of the target in the first image is ( , ), the second pixel coordinate of the target in the second image is ( , ), the moving distance between the moving starting point and the moving end point of the first slider is , the focal length of the monocular camera is .

[0049] Based on the parallax calculation formula, the three-dimensional coordinates of the target ( , , ) in the vertical axis coordinate for:

[0050] The three-dimensional coordinates of the target ( , , ) for:

[0051] The three-dimensional coordinates of the target ( , , ) in the vertical axis for:

[0052] The multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention realizes accurate acquisition of the three-dimensional coordinates of the target at the current observation point through the focal length of the monocular camera, the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of the target.

[0053] In some embodiments, the multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention further includes: In the process of controlling the monocular camera to move from the moving starting point of the first slider to the moving end point of the first slider, each target is continuously tracked based on the multi-target tracking algorithm.

[0054] Specifically, in order to achieve high-precision positioning of the monocular camera during movement, first, a deep neural network is used to identify multiple targets and obtain the pixel coordinates of the target in the current frame. Then, the displacement parameters are set to control the movement of the monocular camera, and the multi-target tracking algorithm is used to keep tracking multiple targets between consecutive frames.

[0055] Figure 4 Schematic diagram of multi-target tracking and positioning provided by the present invention, such as Figure 4As shown, the monocular camera captures a first image at the moving starting point of the first slider, and multi-target recognition is performed on the first image based on the deep neural network, identifying targets 1, 2, and 3. The monocular camera captures a second image at the moving end point of the first slider, and multi-target recognition is performed on the second image based on the deep neural network, and targets 1, 2, and 3 are still identified. In the process of the monocular camera moving from the moving starting point to the moving end point of the first slider, the multi-target tracking algorithm is used to continuously track targets 1, 2, and 3.

[0056] The multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention continuously tracks each target by using a multi-target tracking algorithm during the process of the monocular camera moving from the moving starting point of a first sliding bar to the moving end point, so as to establish an association relationship between the same target in the first image and the second image. By replacing stereo matching with target tracking, the matching error caused by light influence and the like can be reduced, thereby improving the positioning accuracy and stability.

[0057] Figure 5 The overall workflow diagram of the multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention is as follows: Figure 5 As shown, first, a lightweight model is constructed, specifically: a lightweight improvement is made to the deep neural network, and the improved deep neural network is trained using the constructed data set to obtain a trained deep neural network.

[0058] Then, a virtual binocular system is constructed, specifically: a monocular camera captures image data, uses a deep neural network to detect targets and track multiple targets, obtains pixel coordinates of the target, and thus constructs a monocular recognition system. On the basis of the monocular recognition system, the first slider is combined to drive the monocular camera to move on the first slider to construct disparity information, calculate depth information based on the disparity information, and combine the depth information with the pixel coordinates to obtain the three-dimensional coordinate information of the target.

[0059] Finally, on the basis of the virtual pseudo-binocular system, the second slider is combined to drive the virtual binocular system to move on the second slider, thereby constructing a multi-virtual binocular system and realizing multi-point detection.

[0060] In order to facilitate understanding of the multi-target three-dimensional positioning method based on a mobile monocular camera provided by the present invention, a specific embodiment of the present invention is described in detail below.

[0061] The hardware used in the present invention mainly includes: (1) NVIDIA JETSON XAVIER NX as a high-speed visual processing main controller, providing powerful visual and data processing support; (2) industrial monocular camera, which has the advantages of low cost and higher imaging quality; (3) mobile slider. The selection of the above hardware takes into account factors such as processing power, accuracy, power consumption, and volume. Through the use of the above hardware, stable operation and stable positioning of the harvesting target can be achieved.

[0062] When the operating machine, such as a tea harvester, moves to the designated observation point, the monocular camera captures the image data at the current observation point. The tea bud target in the image is identified through the YOLOv8 deep learning network, and the first pixel coordinates of the tea bud target are obtained. At the same time, the distance L of the current monocular camera relative to the starting point on the first slider is saved. c1 . Control the movement of the monocular camera and move the monocular camera to the distance L on the first slider relative to the starting point c2 At this point, we combine mainstream multi-target tracking algorithms such as DeepSort and ByteTrack to always keep continuous tracking of the same tea bud target. Then, we obtain the second pixel coordinate after the position update again.

[0063] Calculate L c2 With L c1 The moving distance between the two objects can be calculated based on the first pixel coordinates and the second pixel coordinates of the tea bud target and the parallax calculation formula to obtain the three-dimensional coordinates of the tea bud target under the virtual binocular system.

[0064] Finally, the virtual binocular system is mounted on the second sliding bar with a larger displacement to drive the entire virtual binocular system to move, and the tea bud targets on the tea shed surface are observed from multiple observation points. For tea bud targets that are repeatedly observed at different points, the spatial distance of the three-dimensional coordinates of the tea bud group obtained by two observations is calculated, and the preset spatial distance Dmin is set. When the spatial distance of the tea buds is less than the preset spatial distance Dmin, they are the same tea buds, otherwise they are two different tea buds. When the spatial distance of the tea buds is less than the preset spatial distance Dmin, it is judged to be the same tea bud, and the three-dimensional coordinates of the two measurements are fused.

[0065] Figure 6 is a schematic diagram of positioning based on a monocular camera provided by the present invention, such as Figure 6 As shown in the figure, the dotted line frame is the observable area of ​​point 1, the solid line frame is the observable area of ​​point 2, the monocular camera observes tea buds 1, 3 and 3 at observation point 1, and the monocular camera observes tea buds 3, 4 and 5 at observation point 2. The three-dimensional coordinates of tea buds 3 observed at observation point 1 are fused with the three-dimensional coordinates of tea buds 3 observed at observation point 2 to obtain the accurate three-dimensional coordinates of tea buds 3.

[0066] The following is a description of the multi-target three-dimensional positioning device based on a mobile monocular camera provided by the present invention. The multi-target three-dimensional positioning device based on a mobile monocular camera described below and the multi-target three-dimensional positioning method based on a mobile monocular camera described above can be referenced to each other.

[0067] Figure 7 : is a schematic diagram of the structure of a multi-target three-dimensional positioning device based on a mobile monocular camera provided by the present invention, such as Figure 7 As shown, the present invention provides a multi-target three-dimensional positioning device based on a mobile monocular camera, which is applied to a mobile mechanism, wherein the mobile mechanism includes a monocular camera, a first sliding rod, a support rod, and a second sliding rod; The monocular camera is arranged on the first sliding bar, and the monocular camera can move horizontally relative to the first sliding bar. The first sliding bar is connected to the second sliding bar through the supporting bar, and the supporting bar can move horizontally relative to the second sliding bar. A plurality of observation points are arranged on the second sliding bar. The movable range of the second sliding bar is greater than the movable range of the first sliding bar. The device comprises: A first control module 710, used for controlling the support rod to move to the observation point of the second sliding rod; The second control module 720 is used to control the monocular camera to move from the moving starting point of the first sliding bar to the moving end point of the first sliding bar when the support bar is at the current observation point, and obtain a first image taken by the monocular camera at the moving starting point of the first sliding bar and a second image taken by the monocular camera at the moving end point of the first sliding bar; A target recognition module 730 is used to perform multi-target recognition on the first image and the second image based on a deep neural network, and obtain a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; An acquisition module 740 is used to acquire the three-dimensional coordinates of each target at the current observation point based on the moving distance between the moving starting point and the moving end point of the first sliding bar and the first pixel coordinates and the second pixel coordinates of each target; The fusion module 750 is used to fuse the three-dimensional coordinates of the same target at multiple observation points while traversing all observation points of the second slider to obtain the three-dimensional fused coordinates of the same target.

[0068] In some embodiments, the fusion module 750 is specifically used to: Calculate the average value of the three-dimensional coordinates of the same target at multiple observation points; The average value is used as the three-dimensional fusion coordinates of the same target.

[0069] In some embodiments, the apparatus further includes a determination module, wherein the determination module is specifically configured to: Calculate the spatial distance between any two targets based on the three-dimensional coordinates of each target at different observation points; Two targets whose spatial distance is less than a preset spatial distance are determined to be the same target.

[0070] In some embodiments, the acquisition module 740 is specifically used to: Acquire the vertical axis coordinate of each target at the current observation point according to the focal length of the monocular camera, the moving distance between the moving starting point and the moving end point of the first sliding bar, and the horizontal coordinate difference between the first pixel coordinate and the second pixel coordinate of each target; Obtaining the horizontal axis coordinate of each target at the current observation point according to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the horizontal coordinate of each target in the first pixel coordinate; The vertical axis coordinate of each target at the current observation point is obtained according to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the vertical coordinate of each target in the first pixel coordinate.

[0071] In some embodiments, the device further comprises a tracking module, wherein the tracking module is specifically configured to: In the process of controlling the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, each target is continuously tracked based on a multi-target tracking algorithm.

[0072] In some embodiments, the deep neural network is YOLOv8.

[0073] It should be noted here that the multi-target three-dimensional positioning device based on a mobile monocular camera provided by the present invention can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0074] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute a multi-target three-dimensional positioning method based on a mobile monocular camera, the method comprising: controlling the support rod to move to the observation point of the second slide bar; when the support rod is at the current observation point, controlling the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, and obtaining a first image taken by the monocular camera at the moving starting point of the first slide bar, and a second image taken by the monocular camera at the moving end point of the first slide bar; performing multi-target recognition on the first image and the second image respectively based on a deep neural network, and obtaining a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; based on the moving distance between the moving starting point and the moving end point of the first slide bar, and the first pixel coordinate and the second pixel coordinate of each target, obtaining the three-dimensional coordinates of each target at the current observation point; when traversing all the observation points of the second slide bar, the three-dimensional coordinates of the same target at multiple observation points are fused to obtain the three-dimensional fused coordinates of the same target.

[0075] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-target three-dimensional positioning method based on a mobile monocular camera provided by the above methods, the method comprising: controlling the support rod to move to the observation point of the second slide bar; when the support rod is at the current observation point, controlling the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, and obtaining a first image taken by the monocular camera at the moving starting point of the first slide bar, and a second image taken by the monocular camera at the moving end point of the first slide bar; performing multi-target recognition on the first image and the second image respectively based on a deep neural network, and obtaining a first pixel coordinate of each target in the first image and a second pixel coordinate in the second image; based on the moving distance between the moving starting point and the moving end point of the first slide bar, and the first pixel coordinate and the second pixel coordinate of each target, obtaining the three-dimensional coordinates of each target at the current observation point; when traversing all observation points of the second slide bar, the three-dimensional coordinates of the same target at multiple observation points are fused to obtain the three-dimensional fused coordinates of the same target.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the multi-target three-dimensional positioning method based on a mobile monocular camera provided by the above-mentioned methods, the method comprising: controlling the support rod to move to the observation point of the second slide bar; when the support rod is at the current observation point, controlling the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, acquiring a first image taken by the monocular camera at the moving starting point of the first slide bar, and a second image taken by the monocular camera at the moving end point of the first slide bar; performing multi-target recognition on the first image and the second image respectively based on a deep neural network, acquiring a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; acquiring the three-dimensional coordinates of each target at the current observation point based on the moving distance between the moving starting point and the moving end point of the first slide bar, and the first pixel coordinates and the second pixel coordinates of each target; while traversing all the observation points of the second slide bar, fusing the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fused coordinates of the same target.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target three-dimensional positioning method based on a mobile monocular camera, characterized in that: Applied to a mobile mechanism, the mobile mechanism comprising a monocular camera, a first sliding bar, a support bar and a second sliding bar; The monocular camera is arranged on the first sliding bar, and the monocular camera can move horizontally relative to the first sliding bar. The first sliding bar is connected to the second sliding bar through the supporting bar, and the supporting bar can move horizontally relative to the second sliding bar. The movable range of the second sliding bar is greater than the movable range of the first sliding bar. A plurality of observation points are arranged within the movable range of the second sliding bar. The method comprises: Controlling the support rod to move to the observation point of the second sliding rod; When the support rod is at the current observation point, control the monocular camera to move from the moving starting point of the first sliding rod to the moving end point of the first sliding rod, and acquire a first image captured by the monocular camera at the moving starting point of the first sliding rod and a second image captured by the monocular camera at the moving end point of the first sliding rod; Based on a deep neural network, perform multi-target recognition on the first image and the second image respectively, and obtain a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; Based on the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of each target, the three-dimensional coordinates of each target at the current observation point are acquired; When all observation points of the second slide bar are traversed, the three-dimensional coordinates of the same target at multiple observation points are fused to obtain the three-dimensional fused coordinates of the same target.

2. The multi-target three-dimensional positioning method based on a mobile monocular camera according to claim 1, characterized in that: The step of fusing the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fused coordinates of the same target includes: Calculate the average value of the three-dimensional coordinates of the same target at multiple observation points; The average value is used as the three-dimensional fusion coordinates of the same target.

3. The multi-target three-dimensional positioning method based on a mobile monocular camera according to claim 2, characterized in that: Before fusing the three-dimensional coordinates of the same target at multiple observation points to obtain the three-dimensional fused coordinates of the same target, the method further includes: Calculate the spatial distance between any two targets based on the three-dimensional coordinates of each target at different observation points; Two targets whose spatial distance is less than a preset spatial distance are determined to be the same target.

4. The multi-target three-dimensional positioning method based on a mobile monocular camera according to claim 1, characterized in that: Based on the moving distance between the moving starting point and the moving end point of the first sliding bar, and the first pixel coordinates and the second pixel coordinates of each target, the three-dimensional coordinates of each target at the current observation point are obtained, including: Acquire the vertical axis coordinate of each target at the current observation point according to the focal length of the monocular camera, the moving distance between the moving starting point and the moving end point of the first sliding bar, and the horizontal coordinate difference between the first pixel coordinate and the second pixel coordinate of each target; Obtaining the horizontal axis coordinate of each target at the current observation point according to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the horizontal coordinate of each target in the first pixel coordinate; The vertical axis coordinate of each target at the current observation point is obtained according to the focal length of the monocular camera, the vertical axis coordinate of each target at the current observation point, and the vertical coordinate of each target in the first pixel coordinate.

5. The multi-target three-dimensional positioning method based on a mobile monocular camera according to claim 1, characterized in that: The method further comprises: In the process of controlling the monocular camera to move from the moving starting point of the first slide bar to the moving end point of the first slide bar, each target is continuously tracked based on a multi-target tracking algorithm.

6. The multi-target three-dimensional positioning method based on a mobile monocular camera according to claim 1, characterized in that: The deep neural network is YOLOv8.

7. A multi-target three-dimensional positioning device based on a mobile monocular camera, characterized in that: Applied to a mobile mechanism, the mobile mechanism comprising a monocular camera, a first sliding bar, a support bar and a second sliding bar; The monocular camera is arranged on the first sliding bar, and the monocular camera can move horizontally relative to the first sliding bar. The first sliding bar is connected to the second sliding bar through the supporting bar, and the supporting bar can move horizontally relative to the second sliding bar. A plurality of observation points are arranged on the second sliding bar. The movable range of the second sliding bar is greater than the movable range of the first sliding bar. The device comprises: A first control module, used for controlling the support rod to move to the observation point of the second sliding rod; a second control module, configured to control the monocular camera to move from a moving starting point of the first sliding bar to a moving end point of the first sliding bar when the support bar is at a current observation point, and to obtain a first image taken by the monocular camera at the moving starting point of the first sliding bar and a second image taken by the monocular camera at the moving end point of the first sliding bar; A target recognition module, used to perform multi-target recognition on the first image and the second image based on a deep neural network, and obtain a first pixel coordinate of each target in the first image and a second pixel coordinate of each target in the second image; an acquisition module, configured to acquire the three-dimensional coordinates of each target at a current observation point based on the moving distance between the moving starting point and the moving end point of the first sliding bar and the first pixel coordinates and the second pixel coordinates of each target; The fusion module is used to fuse the three-dimensional coordinates of the same target at multiple observation points while traversing all observation points of the second sliding bar to obtain the three-dimensional fused coordinates of the same target.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multi-target three-dimensional positioning method based on a mobile monocular camera as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-target three-dimensional positioning method based on a mobile monocular camera as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-target three-dimensional positioning method based on a mobile monocular camera as described in any one of claims 1 to 6 is implemented.

Images