Object position determination method and device and working machine

Abstract

The invention provides an object position determination method and device and an operation machine, and belongs to the technical field of operation machines. The method comprises the following steps: acquiring an image of an object to be measured, wherein the image of the object to be measured is acquired by a binocular camera arranged on an operation machine; determining a key point of the to-be-detected object in the to-be-detected object image to obtain a first coordinate of the key point; determining the depth of the key point based on the first coordinate of the key point, and determining a second coordinate of the key point based on the first coordinate of the key point and the depth of the key point; obtaining a camera pose; and based on the camera pose, converting the second coordinate of the key point into a coordinate of the key point relative to the coordinate system of the operation machine, and obtaining the position of the to-be-measured object. The depth information can be obtained according to the parallax of the image of the to-be-measured object collected by the binocular camera, so that the position information of the to-be-measured object is obtained, and the object position determination precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of construction machinery, and particularly to a method for determining the position of an object, a device for determining the position of an object, a construction machinery, a machine-readable storage medium, and an electronic device. Background Art

[0002] With the development of technology, construction machinery is currently more and more widely used and becoming more and more intelligent. For example, an excavator is one of the important tools for modern engineering construction and production, and is widely used in fields such as earthwork infrastructure construction, water conservancy transportation, and mining excavation. When a construction machinery is in use, it is necessary to determine the position of an object for operation. Taking an excavator as an example, to achieve intelligent assistance or automatic unloading and loading of the excavator into the cargo box of a mining truck, it is necessary to detect the position of the cargo box of the mining truck through sensors.

[0003] When the existing construction machinery determines the position of an object, visual detection is performed through a monocular camera. The monocular camera can complete the visual detection of the object, but it is often difficult to obtain depth information, and thus it is impossible to accurately determine the position of the object. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method for determining the position of an object, a device for determining the position of an object, a construction machinery, a machine-readable storage medium, and an electronic device. The method for determining the position of an object can obtain the position information of the object to be measured and improve the accuracy of determining the position of the object.

[0005] To achieve the above purpose, the first aspect of the present application provides a method for determining the position of an object, including:

[0006] Obtain an image of the object to be measured, where the image of the object to be measured is collected by a binocular camera disposed on the construction machinery;

[0007] Determine the key points of the object to be measured in the image of the object to be measured to obtain the first coordinates of the key points;

[0008] Based on the first coordinates of the key points, determine the depth of the key points, and based on the first coordinates of the key points and the depth of the key points, determine the second coordinates of the key points;

[0009] Obtain the camera pose, where the camera pose is the pose of the binocular camera relative to the construction machinery coordinate system when the binocular camera collects the image of the object to be measured, and the construction machinery coordinate system is a coordinate system established with a specific position or component of the construction machinery as the origin;

[0010] Based on the camera pose, convert the second coordinates of the key points into the coordinates of the key points relative to the construction machinery coordinate system to obtain the position of the object to be measured.

[0011] In the embodiment of the present application, determining the key points of the object to be measured in the image of the object to be measured and obtaining the first coordinates of the key points includes:

[0012] Performing key point detection on the image of the object to be measured by using a preset target detection model to obtain the first coordinates of the key points.

[0013] In the embodiment of the present application, the image of the object to be measured includes a first camera image and a second camera image, and the first coordinates of the key points include the pixel coordinates of the key points in the first camera image and the second camera image;

[0014] Determining the depth of the key points based on the first coordinates of the key points includes:

[0015] Calculating the camera image disparity based on the pixel coordinates of the key points in the first camera image and the second camera image;

[0016] Determining the depth of the key points based on the camera image disparity.

[0017] In the embodiment of the present application, a plurality of angle sensors are arranged on the construction machine, and obtaining the pose of the camera includes:

[0018] Performing forward kinematic analysis on the construction machine to construct a forward kinematic model;

[0019] Obtaining the angle information of the plurality of angle sensors;

[0020] Calculating the pose of the camera based on the angle information of the plurality of angle sensors by using the forward kinematic model.

[0021] A second aspect of the present application provides an object position determination device, including:

[0022] A first acquisition module, configured to acquire an image of an object to be measured, where the image of the object to be measured is acquired by a binocular camera arranged on a construction machine;

[0023] A first determination module, configured to determine the key points of the object to be measured in the image of the object to be measured and obtain the first coordinates of the key points;

[0024] A second determination module, configured to determine the depth of the key points based on the first coordinates of the key points, and determine the second coordinates of the key points based on the first coordinates of the key points and the depth of the key points;

[0025] A second acquisition module, configured to acquire the pose of the camera, where the pose of the camera is the pose of the binocular camera relative to the construction machine coordinate system, and the construction machine coordinate system is a coordinate system established with a specific position or component of the construction machine as the origin;

[0026] A conversion module, configured to convert the second coordinates of the key points into coordinates of the key points relative to the working machine coordinate system based on the camera pose, so as to obtain the position of the object to be measured.

[0027] In an embodiment of the present application, the object to be measured image includes a first camera image and a second camera image, and the first coordinates of the key points include pixel coordinates of the key points in the first camera image and the second camera image;

[0028] The second determination module includes:

[0029] A first calculation unit, configured to calculate a camera image disparity based on the pixel coordinates of the key points in the first camera image and the second camera image;

[0030] A determination unit, configured to determine the depth of the key points based on the camera image disparity.

[0031] A third aspect of the present application provides a working machine, on which a binocular camera is provided, and the working machine determines the position of an object by using the above object position determination method.

[0032] In an embodiment of the present application, the working machine includes a vehicle body and an execution mechanism, one end of the execution mechanism is connected to the vehicle body, and the binocular camera is disposed at one end of the execution mechanism far from the vehicle body.

[0033] A fourth aspect of the present application provides an electronic device, which includes:

[0034] At least one processor;

[0035] A memory connected to the at least one processor;

[0036] Wherein, the memory stores instructions executable by the at least one processor, and the at least one processor realizes the above object position determination method by executing the instructions stored in the memory.

[0037] A fifth aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the processor is configured to execute the above object position determination method.

[0038] Through the above technical solution, an image of an object to be measured is acquired, and the image of the object to be measured is acquired by a binocular camera disposed on a working machine; key points of the object to be measured are determined in the image of the object to be measured to obtain first coordinates of the key points; based on the first coordinates of the key points, depths of the key points are determined, and based on the first coordinates of the key points and the depths of the key points, second coordinates of the key points are determined; a camera pose is acquired, where the camera pose is the pose of the binocular camera relative to a working machine coordinate system when the binocular camera acquires the image of the object to be measured, and the working machine coordinate system is a coordinate system established with a specific position or component of the working machine as the origin; based on the camera pose, the second coordinates of the key points are converted into coordinates of the key points relative to the working machine coordinate system to obtain the position of the object to be measured. Depth information can be obtained according to the parallax of the image of the object to be measured acquired by the binocular camera. Combining the camera imaging principle, the coordinates of the key points in the camera coordinate system can be obtained, and then converted into the working machine coordinate system, so as to obtain the position information of the object to be measured, improving the accuracy of object position determination. Compared with the method of determining the position by point cloud detection using a lidar, the data processing of this solution is simple and the requirement for computing resources is not high. It can not only balance efficiency and accuracy, but also obtain the position information of the object.

[0039] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings are used to provide a further understanding of the embodiments of the present invention, and constitute a part of the specification. Together with the following specific implementation, they are used to explain the embodiments of the present invention, but do not constitute a limitation to the embodiments of the present invention. In the drawings:

[0041] Figure 1 Schematically shows a flowchart of a method for determining an object position according to an embodiment of the present application;

[0042] Figure 2 Schematically shows a schematic diagram of four key points on the upper surface of a mining truck cargo box according to an embodiment of the present application;

[0043] Figure 3 Schematically shows a schematic diagram of a camera installed on an excavator bucket according to an embodiment of the present application;

[0044] Figure 4 Schematically shows a flowchart of position confirmation steps during the intelligent assisted unloading operation of an excavator from a mining truck cargo box according to an embodiment of the present application;

[0045] Figure 5 Schematically shows a schematic diagram of the structure of an object position determination device according to an embodiment of the present application;

[0046] Figure 6 Schematically shows the internal structure diagram of a computer device according to an embodiment of the present application;

[0047] Figure 7 Schematically shows the electrical connection diagram of an excavator according to an embodiment of the present application.

[0048] Description of reference numerals

[0049] 410 - First acquisition module; 420 - First determination module; 430 - Second determination module; 440 - Second acquisition module; 450 - Conversion module; A01 - Processor; A02 - Network interface; A03 - Internal memory; A04 - Display screen; A05 - Input device; A06 - Non-volatile storage medium; B01 - Operating system; B02 - Computer program. Detailed implementation manners

[0050] The following will describe in detail the specific implementation manners of the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention.

[0051] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solution of the present application all comply with the relevant regulations of national laws and regulations. In the embodiments of the present application, some existing industry solutions such as certain software, components, models, etc. may be mentioned. They should be regarded as exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solution of the present application, but it does not mean that the applicant has already or necessarily used this solution.

[0052] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0053] Figure 1A flowchart of a method for determining the position of an object according to an embodiment of the present application is schematically shown. An embodiment of the present application provides a method for determining the position of an object, which is used for a working machine. The working machine may be an intelligent working machine with a sensing system, such as an excavator or a crane. The depth information can be obtained from the parallax of the image of the object to be measured collected by the binocular camera. Combining the camera imaging principle, the coordinates of the key points in the camera coordinate system can be obtained, and then converted to the working machine coordinate system, so as to obtain the position information of the object to be measured, improving the accuracy of object position determination. Compared with the method of determining the position by lidar point cloud detection, the data processing of this solution is simple and the demand for computing resources is not high. It can not only balance efficiency and accuracy, but also obtain the position information of the object. It should be noted that, for the convenience of explaining the solution, in this embodiment, the working machine is mainly an excavator to explain the solution.

[0054] This embodiment provides a method for determining the position of an object, including the following steps:

[0055] Step 210: Obtain an image of the object to be measured, where the image of the object to be measured is collected by a binocular camera arranged on the working machine;

[0056] In this embodiment, the above-mentioned image of the object to be measured can be obtained by adjusting the rotation angle of the working machine so that the binocular camera can collect the object to be measured and then take a picture. In this embodiment, on the premise that the object to be measured can be photographed, the installation position of the binocular camera is not limited. Taking the working machine as an excavator as an example, please refer to Figure 7 , Figure 7 which schematically shows the electrical connection diagram of the excavator according to an embodiment of the present application. By installing sensors to detect the rotation angles of the excavator body, boom, and stick, the binocular camera can be installed on the stick of the excavator. By controlling the angles of the boom and stick, the binocular camera can image the upper surface of the mining truck cargo box. Since the binocular camera includes two cameras, the image of the object to be measured includes two images.

[0057] Step 220: Determine the key points of the object to be measured in the image of the object to be measured to obtain the first coordinates of the key points;

[0058] In this embodiment, the key points of the object to be measured can be determined according to the actual situation, and different objects and different operation scenarios correspond to different key points. The key points can be the key points on the object to be measured, which are used to determine the position of the object. The number of key points can be one or more. Taking a working machine as an excavator as an example, in the operation scenario of the excavator automatically discharging and loading into the cargo box of a mining truck, four vertices on the upper surface of the cargo box of the mining truck can be selected as the key points to determine the position of the cargo box of the mining truck, and then discharging and loading can be carried out. By determining the key points of the object to be measured in the image of the object to be measured, the pixel coordinates of the key points in the image of the object to be measured can be obtained, that is, the first coordinates of the key points are obtained. Since the image of the object to be measured includes two images, correspondingly, the first coordinates of the key points include the coordinates of the key points in the two images respectively.

[0059] In some embodiments, determining the key points of the object to be measured in the image of the object to be measured and obtaining the first coordinates of the key points includes: using a preset target detection model to perform key point detection on the image of the object to be measured to obtain the first coordinates of the key points.

[0060] In this embodiment, the preset target detection model can be pre-trained using a deep learning detection algorithm. By using the target detection model to perform target detection on the image of the object to be measured, the key points are detected, and these key points can be used to locate the position of the object to be measured. Since the image of the object to be measured includes two images, correspondingly, the key point detection is to perform key point detection on the two images respectively.

[0061] Taking a working machine as an excavator as an example, in the operation scenario of the excavator automatically discharging and loading into the cargo box of a mining truck, the construction process of the target detection model includes the following steps:

[0062] First, data construction: Collect pictures of mining trucks of various (different brands, sizes, colors, shapes, and locations), and use rectangular frames in the pictures to label the mining trucks and the four key points on the upper surface of the cargo box. Please refer to Figure 2 , Figure 2 which schematically shows a schematic diagram of four key points on the upper surface of the cargo box of a mining truck according to an embodiment of the present application. Divide all the data sets into a training set, a validation set, and a test set.

[0063] Then, model training: The latest YOLOv11 model can be used to train the target detection model to obtain the optimal weight parameters and save them locally.

[0064] Finally, model deployment: Deploy the trained target detection model to the actual hardware platform, load the optimal weight parameters of the model, and perform key point detection on the images collected by the two cameras for both the mining truck and its cargo box.

[0065] It should be noted that the process of training the object detection model using the above deep learning detection algorithm can be achieved by existing technologies and will not be elaborated here.

[0066] By using the preset object detection model to detect key points of the image of the object to be measured, the key points can be quickly and accurately identified. The latest visual detection model, the YOLOv11 model, is adopted to improve the accuracy and efficiency of detection, which helps to improve the accuracy and efficiency of detection.

[0067] Step 230: Based on the first coordinates of the key points, determine the depth of the key points, and based on the first coordinates of the key points and the depth of the key points, determine the second coordinates of the key points;

[0068] In this embodiment, the coordinates of point p in the camera coordinate system are (X C , Y C , Z C ), and the coordinates in the pixel coordinate system are (u, v). The offset of the origin of the image coordinate system relative to the pixel coordinate system is (u 0 , v 0 ), the camera focal length is f, dx and dy respectively represent how many length units a single pixel in the x and y directions occupies, and the internal parameter matrix of the camera is K, then

[0069]

[0070] According to the above formula, the coordinates of point P in the camera coordinate system can be obtained. However, for a single camera, the depth information of the pixel points cannot be determined, and the depth of the key points can be determined according to the parallax of the images collected by two cameras.

[0071] In some embodiments, the image of the object to be measured includes a first camera image and a second camera image, and the first coordinates of the key points include the pixel coordinates of the key points in the first camera image and the second camera image. Correspondingly, the determining the depth of the key points based on the first coordinates of the key points includes:

[0072] First, based on the pixel coordinates of the key points in the first camera image and the second camera image, calculate the camera image parallax;

[0073] In this embodiment, the above calculation of the camera image parallax can be obtained by subtracting the pixel abscissa of the key point in the first camera image from the pixel abscissa in the second camera image.

[0074] Then, based on the camera image parallax, determine the depth of the key points.

[0075] In this embodiment, taking a construction machine as an excavator as an example, in the operation scenario of automatically unloading and loading the excavator into the cargo box of a mining truck, there are a total of 4 key points of the cargo box. Among them, the depth of the i-th key point can be expressed as

[0076]

[0077] where d i = u Li - u Ri , u Li is the pixel abscissa of the i-th key point in the image collected by the left camera, u Ri is the pixel abscissa of the i-th key point in the image collected by the right camera, and d i is the camera image parallax of the i-th key point, and b is the baseline length of the binocular camera.

[0078] By accurately extracting the pixel coordinates of the key points in different camera images, calculating the camera image parallax, and accurately determining the depth of the key points based on the camera image parallax, it helps to achieve more accurate and reliable visual task processing.

[0079] In this embodiment, after obtaining the first coordinates of the key points and the relative depth of the key points, the coordinates of the key points in the camera coordinate system can be determined to obtain the second coordinates of the key points. Taking a construction machine as an excavator as an example, the binocular camera is installed on the excavator's boom. The coordinates of the origin of the image coordinate system of the right camera in the pixel coordinate system are (u R0 , v R0 ), and the pixel coordinates of the i-th key point in the right camera are (u Ri , v Ri ). Then the pixel coordinate difference between the i-th key point and the origin of the image coordinate system of the right camera is (u Ri - u Ro , v Ri - v Ro ). The coordinates of the i-th key point in the right camera coordinate system are Then:

[0080]

[0081] Step 240: Obtain the camera pose, where the camera pose is the pose of the binocular camera relative to the operation machine coordinate system when the binocular camera collects the image of the object to be measured. The operation machine coordinate system is a coordinate system established with a specific position or component of the operation machine as the origin;

[0082] In this embodiment, in order to accurately obtain the camera pose, the forward kinematic equation can be listed based on the geometric relationship between the camera and the excavator, and then the camera pose can be obtained.

[0083] In some embodiments, a plurality of angle sensors are provided on the work machine, and the obtaining of the camera pose includes:

[0084] First, perform forward kinematic analysis on the work machine to construct a forward kinematic model;

[0085] In this embodiment, the forward kinematics of the work machine can be solved by the geometric method to obtain the forward kinematic model. Taking the work machine as an excavator as an example, the binocular camera can be installed on the excavator's stick. Please refer to Figure 3 , Figure 3 FIG. schematically shows a schematic diagram of a camera installed on the excavator bucket according to an embodiment of the present application. The excavator in the figure includes a body, a boom joint, and a stick joint. The above-mentioned plurality of angle sensors are used to measure the angle information of the body, the boom joint, the stick joint, and the binocular camera. Among them, the body rotation angle is represented as θ 1 , the boom rotation angle is represented as θ 2 , the stick rotation angle is represented as θ 3 , the angle formed by the camera optical axis and the stick is θ 4 . By obtaining multiple joint angles, it is convenient to model from multiple degrees of freedom. The height d from the boom joint to the base 1 ; the lateral distance L from the boom joint to the base 0 ; the distance L from the stick joint to the boom joint 1 ; the distance L from the installation position of the binocular camera to the stick joint 2 ; the baseline length b of the binocular camera; the distance L from the camera optical center to the baseline 3 ; taking the projection of the body rotation center on the ground as the origin, a rectangular coordinate system is established according to the right-hand rule, and the right camera optical center coordinates are represented as (x r , y r , z r ). The angle between the camera optical axis and the horizontal The forward kinematic equations can be listed according to the geometric relationship:

[0086]

[0087] z r = d 1 - (L 1 sinθ 2 + L 2 sin(θ 2 + θ 3 ) + L 3 sin(θ 2 + θ 3 + θ 4 ));

[0088]

[0089] The pose of the binocular camera relative to the working machine coordinate system can be deduced through forward kinematic modeling, which helps to better understand and control the motion behavior of the system.

[0090] Then, the angular information of the multiple angular sensors is obtained;

[0091] In this embodiment, the above-mentioned obtaining the angular information of the multiple angular sensors may be when collecting the image of the object to be measured, and the angular information corresponding to the binocular camera in the current pose is obtained through each sensor.

[0092] Finally, based on the angular information of the multiple angular sensors, the camera pose is calculated using the forward kinematic model.

[0093] In this embodiment, after obtaining the angular information of the multiple angular sensors, substituting the angular information of the multiple angular sensors into the forward kinematic model can calculate the pose of the binocular camera relative to the working machine coordinate system.

[0094] Step 250: Based on the camera pose, convert the second coordinates of the key points into the coordinates of the key points relative to the working machine coordinate system to obtain the position of the object to be measured.

[0095] In this embodiment, according to the camera pose and the imaging principle of the camera, the conversion relationship between the working machine coordinate system and the camera coordinate system can be determined, and then the second coordinates of the key points are converted into the coordinates of the key points relative to the working machine coordinate system by using the conversion relationship, and then the position of the object to be measured is obtained. Taking the working machine as an excavator as an example, the binocular camera is installed on the excavator's dipper arm, and by controlling the angles of the boom and dipper arm, the binocular camera can image the upper surface of the mining truck's cargo box. At the four key points on the upper surface of the cargo box, based on the above example, the coordinates of the four cargo box key points in the excavator coordinate system are respectively (X i , Y i , Z i ), i = 1, 2, 3, 4, then:

[0096]

[0097] Among them, (x r , y r , z r ) is the camera pose; is the second coordinate of the key point, that is, the position of the key point in the camera coordinate system;

[0098] is the formula for rotating around the z-axis in the excavator coordinate system;

[0099] The formula for rotation about the y-axis in the excavator coordinate system;

[0100] It is the conversion relationship between the camera coordinate system and the excavator coordinate system when the excavator does not rotate and the camera optical axis is horizontal and forward.

[0101] According to the above relational expressions, the coordinates (X i , Y i , Z i ) of each key point relative to the excavator coordinate system can be obtained. After determining the coordinates of the four key points relative to the excavator coordinate system, the position of the cargo box can be determined. It should be noted that for extra-large mining trucks, if all four key points of the cargo box cannot be photographed at one time, the rotation angles of the excavator body, boom, and stick can be adjusted, and the key points of the cargo box can be photographed and imaged multiple times.

[0102] In the above implementation process, by acquiring an image of the object to be measured, the image of the object to be measured is acquired by a binocular camera arranged on the working machine; key points of the object to be measured are determined in the image of the object to be measured to obtain the first coordinates of the key points; based on the first coordinates of the key points, the depth of the key points is determined, and based on the first coordinates of the key points and the depth of the key points, the second coordinates of the key points are determined; the camera pose is acquired, where the camera pose is the pose of the binocular camera relative to the working machine coordinate system when the binocular camera acquires the image of the object to be measured, and the working machine coordinate system is a coordinate system established with a specific position or component of the working machine as the origin; based on the camera pose, the second coordinates of the key points are converted into the coordinates of the key points relative to the working machine coordinate system to obtain the position of the object to be measured. According to the parallax of the image of the object to be measured acquired by the binocular camera, depth information can be obtained, and combined with the camera imaging principle, the coordinates of the key points in the camera coordinate system can be obtained, and then converted to the excavator coordinate system, so as to obtain the position information of the object to be measured, improving the accuracy of object position determination. Compared with the method of determining the position by point cloud detection using a lidar, the data processing of this solution is simple, and the demand for computing resources is not high. It can not only balance efficiency and accuracy, but also obtain the position information of the object.

[0103] The following takes the intelligent auxiliary unloading operation of the excavator from the mining truck cargo box as an example to illustrate the solution in detail. Please refer to Figure 4 , Figure 4 which schematically shows the flow chart of the position confirmation steps when the excavator performs intelligent auxiliary unloading operation from the mining truck cargo box according to the embodiment of the present application.

[0104] First, install sensors to detect the angles of the body, boom, and arm of the excavator. The binocular camera is installed on the arm of the excavator. Adjust the pose of the arm so that the binocular camera can image the upper surface of the mining truck cargo box. Then, solve the coordinates of the camera relative to the excavator coordinate system according to the forward kinematics of the camera relative to the excavator coordinate system. Then, train a deep learning model to detect the key points of the cargo box. Obtain the depth of the key points of the cargo box according to the disparity of the images collected by the left and right cameras, and then combine the camera imaging principle to obtain the coordinates of the key points in the camera coordinate system. Finally, based on the coordinates of the camera relative to the excavator and the coordinates of the key points of the cargo box relative to the camera coordinate system, the coordinates of the key points of the cargo box in the excavator coordinate system can be obtained, laying a foundation for the realization of the functions of intelligent auxiliary unloading and automatic unloading of the excavator.

[0105] Figure 1 It is a schematic flowchart of the method for determining the position of an object in the embodiment. It should be understood that although Figure 1 the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, Figure 1 at least a part of the steps in

[0106] This embodiment provides a working machine, on which a binocular camera is provided, and the working machine determines the position of an object by using the above-mentioned method for determining the position of an object.

[0107] In this embodiment, the above-mentioned working machine can be an excavator, a crane, etc. By using the above-mentioned method for determining the position of an object to determine the position of an object, it is possible to obtain the position information of the object by using a binocular camera. This method can obtain depth information according to the disparity of the images of the object to be measured collected by the binocular camera, and combine the camera imaging principle to obtain the coordinates of the key points in the camera coordinate system, and then convert them to the working machine coordinate system, so as to obtain the position information of the object to be measured, improve the accuracy of object position determination, help the working machine to operate accurately, and improve the working efficiency and quality of the working machine.

[0108] In some embodiments, the working machine includes a vehicle body and an actuator, one end of the actuator is connected to the vehicle body, and the binocular camera is disposed at the end of the actuator away from the vehicle body.

[0109] In this embodiment, taking an excavator as an example, the actuating mechanism includes a boom, an arm, a bucket, etc. connected to the vehicle body, and the binocular camera can be arranged on the arm or the bucket.

[0110] By arranging the binocular camera at one end of the actuating mechanism far from the vehicle body, the shooting angle of the binocular camera can be adjusted by controlling the actuating mechanism, so as to obtain a higher and larger field of view, and further improve the accuracy of object position determination.

[0111] Please refer to Figure 5 , Figure 5 which schematically shows a structural diagram of an object position determination device according to an embodiment of the present application. This embodiment provides an object position determination device, including a first acquisition module 410, a first determination module 420, a second determination module 430, a second acquisition module 440, and a conversion module 450, where:

[0112] The first acquisition module 410 is configured to acquire an image of an object to be measured, and the image of the object to be measured is acquired by a binocular camera arranged on a working machine;

[0113] The first determination module 420 is configured to determine key points of the object to be measured in the image of the object to be measured, and obtain a first coordinate of the key points;

[0114] The second determination module 430 is configured to determine a depth of the key points based on the image of the object to be measured and the first coordinate of the key points, and determine a second coordinate of the key points based on the first coordinate of the key points and the depth of the key points;

[0115] The second acquisition module 440 is configured to acquire a camera pose, where the camera pose is a pose of the binocular camera relative to a working machine coordinate system, and the working machine coordinate system is a coordinate system established with a specific position or component of the working machine as the origin;

[0116] The conversion module 450 is configured to convert the second coordinate of the key points into a coordinate of the key points relative to the working machine coordinate system based on the camera pose, so as to obtain the position of the object to be measured.

[0117] Wherein, the image of the object to be measured includes a first camera image and a second camera image, and the first coordinate of the key points includes pixel coordinates of the key points in the first camera image and the second camera image;

[0118] The second determination module 430 includes:

[0119] A first calculation unit, configured to calculate a camera image parallax based on pixel coordinates of the key points in the first camera image and the second camera image;

[0120] A determination unit, configured to determine the key point depth based on the camera image parallax.

[0121] The object position determination device includes a processor and a memory. The above-mentioned first acquisition module 410, first determination module 420, second determination module 430, second acquisition module 440, conversion module 450, etc. are all stored in the memory as program units, and the processor executes the above program units stored in the memory to implement corresponding functions.

[0122] The processor contains a kernel, and the kernel retrieves the corresponding program units from the memory. One or more kernels can be set, and by adjusting the kernel parameters, the position information of the object can be obtained using a binocular camera.

[0123] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash RAM (flash RAM). The memory includes at least one storage chip.

[0124] An embodiment of the present invention provides a machine-readable storage medium, on which a program is stored, and when the program is executed by a processor, the object position determination method is implemented.

[0125] An embodiment of the present invention provides a processor, which is used to run a program, and when the program runs, the object position determination method is executed.

[0126] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 6 shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, an object position determination method is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device A05 of the computer device can be a touch layer covering the display screen, or a button, trackball, or touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0127] Those skilled in the art can understand that Figure 6 The structure shown in Figure 6 is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0128] In one embodiment, the object position determination device provided by the present application can be implemented in the form of a computer program, and the computer program can run on a computer device as shown in Figure 6 In the memory of the computer device, each program module that makes up the object position determination device can be stored. For example, Figure 5 The first acquisition module 410, the first determination module 420, the second determination module 430, the second acquisition module 440, and the conversion module 450 shown in Figure 5 . The computer program composed of each program module enables the processor to execute the steps in the object position determination method of each embodiment of the present application described in this specification.

[0129] Figure 6 The computer device shown in Figure 5 can execute step 210 through the first acquisition module 410 in the object position determination device shown in Figure 5 . The computer device can execute step 220 through the first determination module 420. The computer device can execute step 230 through the second determination module 430. The computer device can execute step 240 through the second acquisition module 440. The computer device can execute step 250 through the conversion module 450.

[0130] An embodiment of the present application provides an electronic device, which includes: at least one processor; a memory connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the at least one processor realizes the above object position determination method by executing the instructions stored in the memory. When the processor executes the instructions, the following steps are realized:

[0131] Obtain an image of the object to be measured, where the image of the object to be measured is collected by a binocular camera disposed on a working machine;

[0132] Determine the key points of the object to be measured in the image of the object to be measured to obtain the first coordinates of the key points;

[0133] Based on the first coordinates of the key points, determine the depth of the key points, and based on the first coordinates of the key points and the depth of the key points, determine the second coordinates of the key points;

[0134] ​​Obtain the camera pose, where the camera pose is the pose of the binocular camera relative to the working machine coordinate system when the binocular camera captures the image of the object to be measured. The working machine coordinate system is a coordinate system established with a specific position or component of the working machine as the origin;

[0135] Based on the camera pose, convert the second coordinate of the key point to the coordinate of the key point relative to the working machine coordinate system to obtain the position of the object to be measured.

[0136] In one embodiment, determining the key points of the object to be measured in the image of the object to be measured to obtain the first coordinates of the key points includes:

[0137] Use a pre-set target detection model to perform key point detection on the image of the object to be measured to obtain the first coordinates of the key points.

[0138] In one embodiment, the image of the object to be measured includes a first camera image and a second camera image, and the first coordinates of the key points include the pixel coordinates of the key points in the first camera image and the second camera image;

[0139] The determining the depth of the key point based on the first coordinates of the key point includes:

[0140] Based on the pixel coordinates of the key point in the first camera image and the second camera image, calculate the camera image parallax;

[0141] Based on the camera image parallax, determine the depth of the key point.

[0142] In one embodiment, a plurality of angle sensors are provided on the working machine, and the obtaining of the camera pose includes:

[0143] Perform forward kinematic analysis on the working machine to construct a forward kinematic model;

[0144] Obtain the angle information of the plurality of angle sensors;

[0145] Based on the angle information of the plurality of angle sensors, use the forward kinematic model to calculate the camera pose.

[0146] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0148] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0150] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0151] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0152] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining the position of an object, characterized in that: include: Acquire an image of the object to be measured, wherein the image of the object to be measured is acquired by a binocular camera arranged on the operating machine; Determine the key points of the object to be measured in the image of the object to be measured, and obtain first coordinates of the key points; Based on the first coordinate of the key point, determine the depth of the key point, and based on the first coordinate of the key point and the depth of the key point, determine the second coordinate of the key point; Acquire a camera pose, wherein the camera pose is the pose of the binocular camera relative to the working machine coordinate system when the binocular camera acquires the image of the object to be measured, and the working machine coordinate system is a coordinate system established with a specific position or component of the working machine as the origin; Based on the camera posture, the second coordinate of the key point is converted into the coordinate of the key point relative to the working machine coordinate system to obtain the position of the object to be measured.

2. The method according to claim 1, characterized in that The step of determining the key points of the object to be measured in the image of the object to be measured and obtaining the first coordinates of the key points includes: A preset target detection model is used to perform key point detection on the image of the object to be detected to obtain the first coordinates of the key points.

3. The method according to claim 1, characterized in that The image of the object to be measured includes a first camera image and a second camera image, and the first coordinates of the key point include pixel coordinates of the key point in the first camera image and the second camera image; The determining the key point depth based on the first coordinate of the key point includes: Calculating the camera image disparity based on the pixel coordinates of the key point in the first camera image and the second camera image; Based on the camera image disparity, the key point depth is determined.

4. The method according to claim 1, characterized in that: The working machine is provided with a plurality of angle sensors, and the obtaining of the camera posture comprises: Performing forward kinematics analysis on the operating machine to construct a forward kinematics model; Acquiring angle information of the multiple angle sensors; Based on the angle information of the multiple angle sensors, the camera pose is calculated using the forward kinematics model.

5. An object position determination device, characterized in that: include: A first acquisition module is used to acquire an image of the object to be measured, wherein the image of the object to be measured is acquired by a binocular camera arranged on the operating machine; A first determination module, used to determine the key points of the object to be measured in the image of the object to be measured, and obtain first coordinates of the key points; A second determination module is used to determine the depth of the key point based on the first coordinate of the key point, and to determine the second coordinate of the key point based on the first coordinate of the key point and the depth of the key point; A second acquisition module is used to acquire a camera posture, wherein the camera posture is the posture of the binocular camera relative to the working machine coordinate system, and the working machine coordinate system is a coordinate system established with a specific position or component of the working machine as an origin; A conversion module is used to convert the second coordinate of the key point into the coordinate of the key point relative to the working machine coordinate system based on the camera posture to obtain the position of the object to be measured.

6. The device according to claim 5, characterized in that The image of the object to be measured includes a first camera image and a second camera image, and the first coordinates of the key point include pixel coordinates of the key point in the first camera image and the second camera image; The second determining module comprises: A first calculation unit, configured to calculate a camera image disparity based on pixel coordinates of the key point in the first camera image and the second camera image; A determination unit is used to determine the depth of the key point based on the camera image parallax.

7. A working machine, characterized in that: The operating machine is provided with a binocular camera, and the operating machine determines the position of an object using the object position determination method described in any one of claims 1 to 4.

8. The working machine according to claim 7, characterized in that: The working machine comprises a vehicle body and an actuator, one end of the actuator is connected to the vehicle body, and the binocular camera is arranged on an end of the actuator away from the vehicle body.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the object position determination method according to any one of claims 1 to 4 by executing the instructions stored in the memory.

10. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the processor is configured to perform the object position determination method according to any one of claims 1 to 4.

Images