Unmanned formula racing obstacle detection method, device, equipment and medium

Abstract

The application relates to an unmanned formula racing car obstacle detection method, device, equipment and medium, the method comprising: performing target detection on a racing track scene image based on a pre-trained cone bucket detection model to determine the two-dimensional frame of each color category of cone bucket in the racing track scene image; assuming the three-dimensional frame of the cone bucket based on the two-dimensional frame of the cone bucket, constructing a projection combination associated with the three-dimensional frame and the two-dimensional frame, constructing an over-determined equation set of associated pixel coordinates and cone bucket three-dimensional coordinates based on the projection points in the projection combination, solving the over-determined equation set by using the least square method to determine the three-dimensional position coordinates of the cone bucket in the camera coordinate system; determining the distance of each color category of cone bucket relative to the unmanned formula racing car based on the three-dimensional position coordinates, and indicating the unmanned formula racing car to adjust the driving direction according to the distance. The application can accurately and quickly avoid the unmanned formula racing car from colliding with the cone bucket obstacle in the racing track.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned vehicles, in particular to an unmanned formula racing car obstacle detection method, a corresponding device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In recent years, with the development of mathematics, sensor technology and computer hardware technology. Under this background, as a key technology in unmanned driving, the accuracy, real-time performance and accuracy of target detection of environmental perception all have important influence on the stability of the whole unmanned driving system. The track is mainly surrounded by conical barrels, among which the left side of the track is red barrels and the right side is blue barrels. There are also large and small yellow barrels at the starting and stopping positions of the racing car.

[0003] At present, for the detection of conical barrels on the unmanned formula racing track, the position of the conical barrel on the two-dimensional image is determined by color and approximate shape. This method can reduce the accuracy of two-dimensional detection of the conical barrel, and further reduce the accuracy of three-dimensional detection. Or use laser point cloud projection to two-dimensional detection to realize the perception of the position of the conical barrel. This kind of algorithm cannot eliminate the influence of background point cloud and conical barrel point cloud projected into the two-dimensional detection frame at the same time, and depends on multi-line laser radar or camera and radar information fusion to obtain the three-dimensional position of the conical barrel, which has high hardware cost and other problems.

[0004] In summary, for the detection of conical barrels on the unmanned formula racing track in the prior art, the two-dimensional detection accuracy of the conical barrel is low, which further reduces the accuracy of three-dimensional detection, and the three-dimensional position of the conical barrel is obtained by relying on multi-line laser radar or camera and radar information fusion, which has high hardware cost and other problems. The present applicant makes corresponding exploration to solve the problem. SUMMARY

[0005] The present application aims to solve the above problems and provide an unmanned formula racing car obstacle detection method, a corresponding device, an electronic device and a computer readable storage medium.

[0006] To achieve the various purposes of the present application, the present application adopts the following technical solutions:

[0007] An unmanned formula racing car obstacle detection method is proposed to adapt to one of the purposes of the present application, comprising:

[0008] In response to the unmanned formula racing car obstacle detection instruction, the track scene image in the monocular camera of the unmanned formula racing car is obtained;

[0009] perform target detection on the race track scene image based on a pre-trained cone detection model to determine a two-dimensional frame of each color category of cone in the race track scene image;

[0010] based on the two-dimensional frame of the cone, assume a three-dimensional frame of the cone, construct a projection combination associated with the three-dimensional frame and the two-dimensional frame, construct an over-determined equation set associated with pixel coordinates and three-dimensional coordinates of the cone based on a projection point pair in the projection combination, solve the over-determined equation set by using a least square method, and determine three-dimensional position coordinates of the cone in a camera coordinate system;

[0011] based on the three-dimensional position coordinates, determine distances of cones of each color category relative to the unmanned formula racing car, and instruct the unmanned formula racing car to adjust a driving direction according to the distances to complete detection of obstacles of the unmanned formula racing car.

[0012] Optionally, the step of training the cone detection model comprises:

[0013] obtain a single training sample and a supervision label thereof in a training set, input the training sample into a preset cone detection model, and extract image feature information of a region corresponding to a coordinate pair labeled by the supervision label in a race track scene image of the training sample;

[0014] classify and map the image feature information to a classification space corresponding to position information and a color category of a plurality of cones, obtain a classification probability corresponding to each classification space, and determine position information and the color category of the cone represented by a classification space with the largest classification probability,

[0015] calculate a loss value corresponding to the position information and the color category of the cone represented by the classification space with the largest classification probability based on the position information and the color category of the cone labeled by the supervision label by using a loss function,

[0016] when each of the loss values reaches a preset threshold value, it is indicated that the cone detection model has been trained to a convergent state, and the training of the cone detection model is completed.

[0017] Optionally, the step of, based on the two-dimensional frame of the cone, assuming a three-dimensional frame of the cone, constructing a projection combination associated with the three-dimensional frame and the two-dimensional frame, and constructing an over-determined equation set associated with pixel coordinates and three-dimensional coordinates of the cone based on a projection point pair in the projection combination, comprises:

[0018] determine eight corner points of the three-dimensional frame of the cone in a world coordinate system with a center point of the three-dimensional frame of the cone in a three-dimensional space as an origin of the world coordinate system;

[0019] Projecting eight corner points of the three-dimensional frame of the cone barrel in the world coordinate system into the camera coordinate system to construct a projection combination of the three-dimensional frame and the two-dimensional frame, and constructing an over-determined equation set of associated pixel coordinates and cone barrel three-dimensional coordinates based on the projection point pairs in the projection combination.

[0020] Optionally, the step of determining the three-dimensional position coordinates of the cone barrel in the camera coordinate system by solving the over-determined equation set by the least square method comprises:

[0021] Solving the over-determined equation set according to multiple projection combinations of the three-dimensional frame and the two-dimensional frame by the least square method;

[0022] Projecting the three-dimensional frame center of the cone barrel obtained from each projection combination onto the two-dimensional image to determine the projection point, comparing the projection point with the two-dimensional frame center point of the cone barrel, and selecting the solution corresponding to the projection combination with the smallest pixel distance between the two points as the three-dimensional position coordinates of the cone barrel.

[0023] Optionally, the cone barrels of each color category include one or any multiple of red cone barrels, blue cone barrels, and yellow cone barrels.

[0024] The red cone barrels represent obstacles on the left side of the unmanned formula racing track, the blue cone barrels represent obstacles on the right side of the unmanned formula racing track, and the yellow cone barrels represent the starting point or the end point of the unmanned formula racing track.

[0025] Optionally, the step of instructing the unmanned formula racing car to adjust the driving direction according to the distance of each color category of the cone barrel relative to the unmanned formula racing car based on the three-dimensional position coordinates comprises:

[0026] Determining the three-dimensional position coordinates of each color category of the cone barrel relative to the monocular camera, and determining the distance of each color category of the cone barrel relative to the unmanned formula racing car according to the three-dimensional position coordinates;

[0027] Detecting whether the distance is less than a preset collision distance, and if so, instructing the unmanned formula racing car to adjust the driving direction based on the cone barrels of different colors to avoid collision of the unmanned formula racing car.

[0028] Optionally, the basic network architecture of the cone barrel detection model is a YoloV7 model.

[0029] Another object of the present application is to provide an unmanned formula racing car obstacle detection device, comprising:

[0030] A track image acquisition module is configured to acquire a track scene image in a monocular camera of an unmanned formula racing car in response to an unmanned formula racing car obstacle detection instruction.

[0031] A cone two-dimensional frame determination module is configured to perform target detection on the racing track scene image based on a pre-trained cone detection model to determine a two-dimensional frame of each color category of cone in the racing track scene image;

[0032] An overdetermined equation set determination module is configured to assume a three-dimensional frame of the cone based on the two-dimensional frame of the cone, construct a projection combination associated with the two-dimensional frame and the three-dimensional frame, and construct an overdetermined equation set of associated pixel coordinates and cone three-dimensional coordinates based on a projection point pair in the projection combination;

[0033] A three-dimensional coordinate determination module is configured to solve the overdetermined equation set by using a least square method to determine a three-dimensional position coordinate of the cone in a camera coordinate system;

[0034] An obstacle detection module is configured to determine a distance of each color category of cone relative to the unmanned formula racing car based on the three-dimensional position coordinate, and instruct the unmanned formula racing car to adjust a driving direction according to the distance to complete the detection of the obstacle of the unmanned formula racing car.

[0035] Another object of the present application is to provide an electronic device comprising a central processing unit and a memory, wherein the central processing unit is configured to invoke a computer program stored in the memory to execute the steps of the method for detecting the obstacle of the unmanned formula racing car.

[0036] Another object of the present application is to provide a computer readable storage medium storing a computer program implemented according to the method for detecting the obstacle of the unmanned formula racing car in the form of computer readable instructions, wherein the computer program is invoked and run by a computer to execute the steps included in the corresponding method.

[0037] Compared with the prior art, the present application aims to solve the problems of the prior art, such as low accuracy of two-dimensional detection of cones on the racing track, which further reduces the accuracy of three-dimensional detection, and high hardware cost of obtaining the three-dimensional position of the cone by relying on multi-line laser radar or information fusion of cameras and radars. The present application includes but is not limited to the following beneficial effects:

[0038] The present application obtains the image coordinates of the two-dimensional frame of the cone by obtaining the monocular camera information and the prior size information of the cone, makes an assumption of the three-dimensional frame based on the two-dimensional frame, converts the spatial coordinate system, and finally establishes the three-dimensional spatial coordinate system to solve the position of the cone. In the case of strong priori, the present application realizes the monocular three-dimensional target detection of the cone, and has the characteristics of low hardware cost, high accuracy and stable recognition;

[0039] Further, the unmanned formula racing car obstacle detection method of the present application can avoid the unmanned formula racing car from colliding with the cone obstacles in the track, greatly improve the reaction of the unmanned formula racing car to avoid the cone obstacles of various color categories, significantly improve the driving speed of the unmanned formula racing car, ensure the rapid and orderly progress of the unmanned formula racing competition, and greatly improve the competition results of the unmanned formula racing car. BRIEF DESCRIPTION OF DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0041] Figure 1 An exemplary network architecture used by the unmanned formula racing car obstacle detection method of the present application;

[0042] Figure 2 A schematic diagram of the world coordinate system construction and two-dimensional frame pixel coordinates in the embodiments of the present application;

[0043] Figure 3 A schematic diagram of the three-dimensional detection frame in the embodiments of the present application;

[0044] Figure 4 A schematic diagram of the cone three-dimensional position bird's eye view in the embodiments of the present application;

[0045] Figure 5 A principle block diagram of the unmanned formula racing car obstacle detection device in the embodiments of the present application;

[0046] Figure 6 A structural schematic diagram of the computer device in the embodiments of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, in which the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation of the present application.

[0048] It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, "connected," "coupled," and / or "coupling," can include both direct connections and / or indirect connections (i.e., via one or more other elements). As used herein, "connection" or "coupling" can include a wireless connection or a wireless coupling. As used herein, the term "and / or" comprises all of the associated listed items, one or more of the associated listed items, and all combinations of the associated listed items.

[0049] It will be further understood that, as used herein, the terms "comprises" and / or "comprising," while they can be used in the context of compositions of matter, preferably, are not used in a restrictive sense, for example, to denote the presence of stated features, integers, steps or components thereof, but to denote the presence at least the stated features, integers, steps or components thereof. As used herein, unless otherwise stated, the term "or" includes both exclusive and inclusive or. As used herein, unless otherwise stated, the term "and / or" includes all of the associated listed items, one or more of the associated listed items, and all combinations of the associated listed items.

[0050] Those skilled in the art will understand that, as used herein, the terms "client," "terminal," and "terminal device" include both devices that are solely wireless signal receivers and devices that have both receiving and transmitting hardware that can communicate bi-directionally over a bi-directional communication link. Such devices can include cellular or other communication devices with single-line or multiple-line displays, or no display, Personal Communications Service (PCS) devices that can combine a voice and / or data processor, a PDA that can include a radio frequency receiver and a pager, Internet and / or Intranet access, a Web browser, a calendar, and / or a GPS receiver, a conventional laptop and / or palmtop computer and / or other devices that have a radio frequency receiver. As used herein, the terms "client," "terminal," and "terminal device" can be portable, transportable, mounted in a vehicle (aeronautical, maritime, and / or land), or adapted and / or configured for local and / or distributed operation on Earth and / or any other location in space. As used herein, the terms "client," "terminal," and "terminal device" can also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a Mobile Internet Device (MID), and / or a mobile phone with music / video playing function, a smart television, a set-top box, and / or the like.

[0051] As used herein, the terms "server," "client," "service node," and the like refer to hardware that has the equivalent capability of a personal computer, i.e., an electronic device having a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, and the like necessary components disclosed by the Von Neumann principle. A computer program is stored in the memory, the central processing unit loads the program stored in the external memory into the memory and runs it, executes the instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

[0052] It should be noted that the concept of "server" in the present application can also be extended to the case of a server cluster. According to the principle of network deployment understood by those skilled in the art, the servers should be logically divided, and in physical space, these servers can be independent of each other but can be called through an interface, or can be integrated into a physical computer or a computer cluster. Those skilled in the art should understand this variation and should not be restricted by the implementation of the network deployment of the present application.

[0053] One or more technical features of the present application, unless explicitly specified, can be deployed on a server for implementation and accessed by a client remotely calling an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0054] The neural network model referred to or possibly referred to in the present application, unless explicitly specified, can be deployed on a remote server and remotely called by a client, or can be deployed on a client with sufficient device capability for direct calling. In some embodiments, when it is run on a client, its corresponding intelligence can be obtained through transfer learning, so as to reduce the requirement for client hardware running resources and avoid excessive occupation of client hardware running resources.

[0055] Unless explicitly specified, the various data involved in the present application can be stored remotely on a server or stored locally on a terminal device, as long as it is suitable for being called by the technical solutions of the present application.

[0056] Those skilled in the art should know that the various methods of the present application, although based on the same concept and described to present commonality among them, are independently executable unless otherwise specified. Similarly, for each embodiment disclosed in the present application, it is based on the same inventive concept, so the same concept is understood to be equivalent, and although the concept is expressed differently, it is only a suitable transformation for convenience.

[0057] Unless it is explicitly stated that the various embodiments disclosed in the present application are mutually exclusive, the related technical features involved in each embodiment can be cross-combined to flexibly construct new embodiments, as long as such combination does not deviate from the spirit of the present application and can meet the needs of the prior art or solve some aspects of the deficiencies in the prior art. For this variation, those skilled in the art should know.

[0058] In recent years, with the development of mathematics, sensor technology and computer hardware technology. In this context, the Society of Automotive Engineers of China began to hold the China University Student Competition (FSAC) in 2017. As a key technology in autonomous driving, the accuracy, real-time performance and accuracy of the cone bucket of target detection all have an important influence on the stability of the subsequent autonomous driving system as a whole. The track of FSAC is mainly surrounded by conical cone buckets, among which the left side of the track is a red cone bucket and the right side is a blue cone bucket. There are also large and small yellow cone buckets at the starting and stopping positions of the racing car.

[0059] Based on the reference to the above exemplary scenarios, please refer to Figure 1 The obstacle detection method for the autonomous racing car of the present application comprises, in one embodiment thereof:

[0060] Step S10, in response to the obstacle detection instruction of the autonomous racing car, acquiring the track scene image in the monocular camera of the autonomous racing car;

[0061] The terminal device of the autonomous racing car can respond to the obstacle detection instruction of the autonomous racing car to acquire the track scene image in the monocular camera of the autonomous racing car, which contains roads, various color categories of cone buckets, etc.; the various color categories of cone buckets include one or any number of red cone buckets, blue cone buckets and yellow cone buckets; the red cone bucket represents the obstacle on the left side of the autonomous racing track, the blue cone bucket represents the obstacle on the right side of the autonomous racing track, and the yellow cone bucket represents the starting point or the end point of the autonomous racing track.

[0062] In some embodiments, after acquiring the track scene image in the monocular camera of the autonomous racing car, data augmentation can be performed on the track scene image. The track scene image is subjected to noise addition, partial deletion, rotation, cropping and translation operations for data augmentation to improve the accuracy and robustness of the cone bucket detection model for target detection of the track scene image, and the image after data augmentation is labeled in a semi-automatic manner, which can greatly improve the data labeling efficiency.

[0063] In some embodiments, the sample is processed by randomly selecting at least one data enhancement method such as adding noise, deleting part, rotating, cropping and translating operation, etc., and each sample of the original data set is expanded to ten samples, obtaining 4410 new samples, and finally obtaining 4410 labeled sample pictures through semi-automatic labeling. For the samples that are not well labeled by the semi-automatic tool, 3528 samples are randomly divided as the training set and 882 samples are randomly divided as the verification set according to a 7:3 ratio. After data enhancement, the data amount for training is increased, and the generalization ability of the model is improved; the noise data is increased, the robustness of the two-dimensional detection model is improved, and more accurate two-dimensional detection frame is provided for three-dimensional target detection.

[0064] Step S20, performing target detection on the race track scene image based on the pre-trained cone detection model to determine the two-dimensional frame of each color category of cone in the race track scene image;

[0065] After obtaining the race track scene image in the monocular camera of the unmanned formula racing car, performing target detection on the race track scene image based on the pre-trained cone detection model to determine the two-dimensional frame of each color category of cone in the race track scene image, such as red cone, blue cone and yellow cone.

[0066] The step of training the cone detection model comprises:

[0067] Step S201, obtaining a single training sample and its supervision label in the training set, inputting the training sample into a preset cone detection model, and extracting image feature information of a region corresponding to coordinates labeled by the supervision label in a race track scene image of the training sample;

[0068] Step S203, classifying and mapping the image feature information to a classification space corresponding to the position information and color category of the plurality of cones, obtaining a classification probability corresponding to each classification space, and determining the position information and color category of the cone represented by the classification space with the maximum classification probability;

[0069] Step S205, using a loss function to calculate a loss value corresponding to the position information and color category of the cone represented by the classification space with the maximum classification probability based on the position information and color category of the cone labeled by the supervision label;

[0070] Step S207, when each of the loss values reaches a preset threshold value, it indicates that the cone detection model has been trained to a converged state, thereby completing the training of the cone detection model.

[0071] Specifically, the base network architecture of the cone detection model can be YoloV7 model, etc. The track scene images of cones of various color categories can be pre-collected as training samples, which can be derived from historical video frames or images in a Formula Racing event, etc. The training samples are labeled with their supervision labels. The position information of the cones in the track scene images of the training samples and the color categories of the cones are labeled. Based on the labeling of each training sample, the training set can be constructed by mapping each training sample and its supervision label. The single training sample and its supervision label in the training set are obtained. The training sample is input into the preset cone detection model. The image feature information of the region corresponding to the coordinates labeled by the corresponding supervision label in the track scene image of the training sample is extracted. The image feature information is classified and mapped to the classification space corresponding to the position information and color categories of the plurality of cones. The classification probability corresponding to each classification space is obtained. The position information and color categories of the cones represented by the classification space with the maximum classification probability are determined. The loss function is used to calculate the loss value corresponding to the position information and color categories of the cones represented by the classification space with the maximum classification probability based on the position information and color categories of the cones labeled by the supervision label. When each of the loss values reaches a preset threshold, it indicates that the cone detection model has been trained to a convergent state, thereby completing the training of the cone detection model. Otherwise, it indicates that the model has not converged. The model can be updated according to the gradient of each loss value. The weights and parameters of each part of the model are usually corrected by backpropagation to make the model further approach convergence. Then, the next training sample in the training set is called to implement iterative training on the model until the model is trained to a convergent state. It is not difficult to understand that the cone detection model trained to convergence can detect the position information and color categories of the cones in the current track scene image in the game picture, and determine the two-dimensional frame of the cones of various color categories in the track scene image.

[0072] Step S30, based on the two-dimensional frame of the cone, the three-dimensional frame of the cone is assumed, the projection combination associated with the two-dimensional frame and the three-dimensional frame is constructed, the overdetermined equation set of associated pixel coordinates and cone three-dimensional coordinates is constructed based on the projection point pair in the projection combination, the least squares method is used to solve the overdetermined equation set, and the three-dimensional position coordinates of the cone in the camera coordinate system are determined.

[0073] After determining the two-dimensional frame of each color category of the cone, such as the red cone, the blue cone and the yellow cone in the track scene image, a three-dimensional frame of the cone is assumed based on the two-dimensional frame of each color category of the cone, a projection combination associated with the three-dimensional frame and the two-dimensional frame is constructed, and an over-determined equation set of associated pixel coordinates and cone three-dimensional coordinates is constructed based on the projection point pairs in the projection combination.

[0074] The step of assuming the three-dimensional frame of the cone based on the two-dimensional frame of the cone, constructing a projection combination associated with the three-dimensional frame and the two-dimensional frame, and constructing an over-determined equation set of associated pixel coordinates and cone three-dimensional coordinates based on the projection point pairs in the projection combination comprises:

[0075] In step S301, the center point of the three-dimensional frame of the cone in the three-dimensional space is taken as the origin of the world coordinate system, and eight corner points of the three-dimensional frame of the cone in the world coordinate system are determined.

[0076] In step S303, the eight corner points of the three-dimensional frame of the cone in the world coordinate system are projected into the camera coordinate system to construct a projection combination of the three-dimensional frame and the two-dimensional frame, and an over-determined equation set of associated pixel coordinates and cone three-dimensional coordinates is constructed based on the projection point pairs in the projection combination.

[0077] Further, the step of determining the three-dimensional position coordinates of the cone in the camera coordinate system by solving the over-determined equation set by the least square method comprises:

[0078] In step S3001, the over-determined equation set is solved by the least square method according to the plurality of projection combinations of the three-dimensional frame and the two-dimensional frame.

[0079] In step S3003, the center of the three-dimensional frame of the cone obtained by each projection combination is projected onto the two-dimensional image to determine the projection point, the projection point is compared with the center point of the two-dimensional frame of the cone, and the solution corresponding to the projection combination with the smallest pixel distance between the two points is selected as the three-dimensional position coordinates of the cone.

[0080] Specifically, based on the two-dimensional frames of the cone barrels of various colors, such as the red cone barrel, the blue cone barrel, and the yellow cone barrel, the three-dimensional frame of the cone barrel is assumed. Combined with the prior scene, 64 projection combinations of eight corner points and pixel coordinates of the three-dimensional frame can be constructed; in order to simplify the calculation, the center point of the three-dimensional frame of the cone barrel of various colors, such as the red cone barrel, the blue cone barrel, and the yellow cone barrel in the three-dimensional space is used as the origin of the world coordinate system, and the length, width, and height of the target are used as the directions of the three coordinates of the world coordinate system, and the size of the prior target is used as the length, width, and height of the three-dimensional frame, where dx, dy, and dz are the width, height, and length of the cone barrel, respectively. Then, the eight corner points of the cone barrels of various colors, such as the red cone barrel, the blue cone barrel, and the yellow cone barrel in the world coordinate system are:

[0081]

[0082] Furthermore, based on the projected point pairs in the projection combination, an overdetermined set of equations associating pixel coordinates and cone three-dimensional coordinates is constructed. The specific steps are as follows:

[0083] Projecting from world coordinates to camera coordinates:

[0084]

[0085] In formula (1), X c is the x coordinate in the camera coordinate system, X w is the x coordinate in the world coordinate system, R 3×3 、T 3×1 are the rotation and translation matrices respectively;

[0086] From camera coordinate system to pixel coordinate system:

[0087]

[0088] In formula (2), u and v represent pixel coordinates, dx and dy represent pixel sizes, and f represents the focal length of the camera. Combining the above formulas (1) and (2), it can be expressed as follows:

[0089]

[0090] In formula (3), K is the camera intrinsic parameter matrix, which can be obtained by calibrating the camera intrinsic parameters.

[0091] Formula (3) can be transformed into the following:

[0092]

[0093] R 3×3 and [X w Y w Z w ]T As the known quantities, the total known quantities are combined and replaced by the conversion matrix M to obtain formula (5), which is expressed as follows.

[0094]

[0095] From formula (5), the Z direction coordinate Z of the point in the camera coordinate system can be expressed as: c

[0096]

[0097] Substituting formula 6 into the above formula (5), formulas (7) and (8) for pixel coordinates u and v, respectively, can be obtained, which are expressed as follows:

[0098]

[0099] Referring to Figure 2 , Figure 3 and Figure 4 , based on the two-dimensional frame of the barrel, the three-dimensional frame of the barrel is assumed, and the projection combination associated with the two-dimensional frame is constructed, 64 projection combinations can be constructed, each combination contains 4 projection point pairs, and the four coordinate values u min ,u max ,v min ,v max of the barrel two-dimensional detection frame can be extracted Figure 2 As shown in the figure, the four coordinate values u min ,u max ,v min ,v max of the barrel two-dimensional detection frame are the pixel coordinate values (u min ,v min ) of the left upper corner and the right lower corner of the two-dimensional frame (u max ,v max ) respectively. By substituting them into formula (7) and formula (8), the over-determined equation group (9) for unknown numbers with 4 equations can be constructed, and T 3×1 (the translation matrix of the world coordinate system and the camera coordinate system) can be obtained by the least square method. Because the world coordinate system is established on the three-dimensional center of the target, the solved T 3×1 is the x, y, z coordinate position of the camera relative to the target. Therefore, -T 3×1 is the three-dimensional position information of the target in the camera coordinate system, and the over-determined equation group (9) is expressed as follows:

[0100]

[0101] 64 projection combinations solve 64-T 3×1 ​projecting the center of the target three-dimensional frame obtained by each combination onto a two-dimensional image, comparing the projection point with the center point of the target two-dimensional frame obtained by the cone bucket detection model, and selecting the projection combination corresponding to the minimum pixel distance between the two points 3×1 a three-dimensional position coordinate as a target.

[0102] Step S50, determining the distance of each color category of the cone bucket relative to the unmanned formula racing car based on the three-dimensional position coordinate, and instructing the unmanned formula racing car to adjust the driving direction according to the distance to complete the detection of the unmanned formula racing car obstacle.

[0103] After determining the three-dimensional position coordinate of the cone bucket in the camera coordinate system, the distance of each color category of the cone bucket relative to the unmanned formula racing car is determined based on the three-dimensional position coordinate, and the driving direction of the unmanned formula racing car is adjusted according to the distance to complete the detection of the unmanned formula racing car obstacle.

[0104] In some embodiments, the step of determining the distance of each color category of the cone bucket relative to the unmanned formula racing car based on the three-dimensional position coordinate and instructing the unmanned formula racing car to adjust the driving direction according to the distance comprises:

[0105] Step S501, determining the three-dimensional position coordinate of each color category of the cone bucket relative to the monocular camera, and determining the distance of each color category of the cone bucket relative to the unmanned formula racing car according to the three-dimensional position coordinate;

[0106] Step S503, detecting whether the distance is less than a preset collision distance, and if so, instructing the unmanned formula racing car to adjust the driving direction based on the cone bucket of different colors to avoid collision of the unmanned formula racing car.

[0107] Specifically, after determining the three-dimensional position coordinates of each color category of the cone, the red cone represents the obstacle on the left side of the unmanned formula racing track, the blue cone represents the obstacle on the right side of the unmanned formula racing track, and the yellow cone represents the starting point or the end point of the unmanned formula racing track, and the distance of each color category of the cone from the unmanned formula racing car is determined according to the three-dimensional position coordinates of each color category of the cone; whether the distance of each color category of the cone from the unmanned formula racing car is less than the preset collision distance is detected, and if so, the unmanned formula racing car is instructed to adjust the driving direction based on the different color of the cone, when the cone is blue, the unmanned formula racing car is instructed to adjust the driving direction to the left, and when the cone is red, the unmanned formula racing car is instructed to adjust the driving direction to the right, so as to avoid collision between the unmanned formula racing car and the cone, and ensure the orderly progress of the competition.

[0108] From the above embodiment, compared with the prior art, the present application aims at the problems in the prior art, such as low accuracy of two-dimensional detection of the cone on the unmanned formula racing track, which further reduces the accuracy of three-dimensional detection, and high hardware cost of obtaining the three-dimensional position of the cone by relying on multi-line laser radar or information fusion of camera and radar. The present application includes but is not limited to the following beneficial effects:

[0109] The present application obtains the image coordinates of the two-dimensional frame of the cone by obtaining the monocular camera information and the prior size information of the cone, makes an assumption of three-dimensional frame according to the two-dimensional frame, converts the spatial coordinate system, and finally establishes a three-dimensional spatial coordinate system to solve the position of the cone. In the case of strong priori, the monocular three-dimensional target detection of the cone is realized, and the present application has the characteristics of low hardware cost, high accuracy and stable recognition;

[0110] Further, the unmanned formula racing car obstacle detection method of the present application can accurately and quickly detect the position and color category of each color category of the cone obstacle, avoid the unmanned formula racing car from colliding with the cone obstacle in the racing track, greatly improve the reaction of the unmanned formula racing car to avoid each color category of the cone obstacle, significantly improve the driving speed of the unmanned formula racing car, ensure the rapid and orderly progress of the unmanned formula racing competition, and greatly improve the competition result of the unmanned formula racing car.

[0111] Please refer to Figure 5, provided by one of the purposes of the application, an unmanned formula racing car obstacle detection device, comprising a track image acquisition module 1100, a cone bucket two-dimensional frame determination module 1200, an overdetermined equation set determination module 1300, a three-dimensional coordinate determination module 1400, and an obstacle detection module 1500. Among them, the track image acquisition module 1100 is set to respond to the unmanned formula racing car obstacle detection instruction to acquire the track scene image in the monocular camera of the unmanned formula racing car; the cone bucket two-dimensional frame determination module 1200 is set to perform target detection on the track scene image based on the pre-trained cone bucket detection model to determine the two-dimensional frame of each color category of the cone bucket in the track scene image; the overdetermined equation set determination module 1300 is set to assume the three-dimensional frame of the cone bucket based on the two-dimensional frame of the cone bucket, construct the projection combination associated with the three-dimensional frame and the two-dimensional frame, and construct the overdetermined equation set associated with the pixel coordinates and the three-dimensional coordinates of the cone bucket based on the projection points in the projection combination; the three-dimensional coordinate determination module 1400 is set to solve the overdetermined equation set by using the least square method to determine the three-dimensional position coordinates of the cone bucket in the camera coordinate system; the obstacle detection module 1500 is set to determine the distance of each color category of the cone bucket relative to the unmanned formula racing car based on the three-dimensional position coordinates, and instruct the unmanned formula racing car to adjust the driving direction according to the distance to complete the detection of the obstacles of the unmanned formula racing car.

[0112] On the basis of any embodiment of the present application, please refer to Figure 6 Another embodiment of the present application also provides an electronic device, which can be realized by a computer device, as shown in Figure 6 The internal structure diagram of the computer device. The computer device includes a processor, a computer readable storage medium, a memory and a network interface connected by a system bus. Among them, the computer readable storage medium of the computer device stores an operating system, a database and a computer readable instruction, the database can store control information sequence, and the computer readable instruction can make the processor realize an unmanned formula racing car obstacle detection method when executed by the processor. The processor of the computer device is used to provide computing and control capability to support the operation of the entire computer device. The memory of the computer device can store computer readable instructions, which can make the processor execute the unmanned formula racing car obstacle detection method of the present application when executed by the processor. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0113] The processor in this embodiment is configured to execute the specific functions of each module and its sub-modules in Figure 5 The memory stores the program codes and various data required for executing the above modules or sub-modules. The network interface is configured to transmit data between the user terminal and the server. The memory in this embodiment stores the program codes and data required for executing all modules / sub-modules in the unmanned racing car obstacle detection device of the present application, and the server can call the program codes and data of the server to execute the functions of all sub-modules.

[0114] The present application also provides a storage medium storing computer readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the unmanned racing car obstacle detection method described in any embodiment of the present application.

[0115] The present application also provides a computer program product, including computer programs / instructions, which, when executed by one or more processors, implement the steps of the unmanned racing car obstacle detection method described in any embodiment of the present application.

[0116] It is understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments of the present application can be completed by a computer program instructing related hardware, which can be stored in a computer readable storage medium. The program, when executed, can include the processes of the above-mentioned embodiments of each method. The storage medium can be a computer readable storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0117] The above only describes some embodiments of the present application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered within the scope of protection of the present application.

[0118] In summary, the unmanned racing car obstacle detection method of the present application can accurately and quickly detect the position and color of each color of the cone obstacle, avoid the unmanned racing car from colliding with the cone obstacle in the track, greatly improve the reaction of the unmanned racing car to avoid each color of the cone obstacle, significantly improve the driving speed of the unmanned racing car, ensure the rapid and orderly progress of the unmanned racing car competition, and greatly improve the competition results of the unmanned racing car.

Claims

1. A method for detecting obstacles in an unmanned formula car, characterized in that: include: Responding to the obstacle detection command of the unmanned formula car, obtaining a track scene image from the monocular camera of the unmanned formula car; Performing target detection on the track scene image based on a pre-trained cone detection model to determine the two-dimensional frames of cones of various colors in the track scene image; Based on the two-dimensional frame of the cone barrel, a three-dimensional frame of the cone barrel is assumed, a projection combination associated with the three-dimensional frame and the two-dimensional frame is constructed, and based on the projection point pairs in the projection combination, an overdetermined equation group associating pixel coordinates and the three-dimensional coordinates of the cone barrel is constructed, which includes: Taking the center point of the three-dimensional frame of the cone barrel in the three-dimensional space as the origin of the world coordinate system, and determining the eight corner points of the three-dimensional frame of the cone barrel in the world coordinate system; Projecting the eight corner points of the three-dimensional frame of the cone barrel in the world coordinate system into the camera coordinate system to construct a projection combination of the three-dimensional frame and the two-dimensional frame, and constructing an overdetermined set of equations relating pixel coordinates and the three-dimensional coordinates of the cone barrel based on the projection point pairs in the projection combination; The overdetermined set of equations is solved using the least squares method to determine the three-dimensional position coordinates of the cone barrel in the camera coordinate system, which includes: Solving the overdetermined system of equations based on multiple projection combinations of the three-dimensional box and the two-dimensional box using a least squares method; The center of the cone barrel's 3D frame obtained by each projection combination is projected onto the 2D image to determine the projection point, the projection point is compared with the center point of the cone barrel's 2D frame, and the solution corresponding to the projection combination with the smallest pixel distance between the two points is selected as the 3D position coordinate of the cone barrel; The distances of the cone barrels of each color relative to the driverless formula racing car are determined based on the three-dimensional position coordinates, and the driverless formula racing car is instructed to adjust its driving direction according to the distances to complete the obstacle detection of the driverless formula racing car.

2. The obstacle detection method for an unmanned formula car according to claim 1, characterized in that: The steps to train the cone bucket detection model include: Obtain a single training sample and its supervisory label from the training set, input the training sample into a preset cone detection model, and extract image feature information of the area corresponding to the coordinates marked by the corresponding supervisory label in the track scene image of the training sample; The image feature information is classified and mapped to a preset classification space corresponding to the position information and color types of multiple cone barrels, and the classification probability corresponding to each classification space is obtained. The position information of the cone barrel and the color type of the cone barrel represented by the classification space with the largest classification probability are determined. A loss function is used to calculate the loss value corresponding to the position information of the cone bucket and the color type of the cone bucket represented by the supervision label, which is the classification space with the maximum classification probability. When the loss values ​​of each item reach the preset threshold, it indicates that the cone bucket detection model has been trained to a convergence state, thereby completing the training of the cone bucket detection model.

3. The obstacle detection method for an unmanned formula car according to claim 1, characterized in that: The cone barrels of each color include one or more of a red cone barrel, a blue cone barrel and a yellow cone barrel; The red cone barrel represents the obstacle on the left side of the driverless formula track, the blue cone barrel represents the obstacle on the right side of the driverless formula track, and the yellow cone barrel represents the starting point or end point of the driverless formula track.

4. The obstacle detection method for an unmanned formula car according to claim 3, characterized in that: The steps of determining the distances of the cone barrels of each color relative to the driverless formula racing car based on the three-dimensional position coordinates, and instructing the driverless formula racing car to adjust its driving direction according to the distances include: Determining the three-dimensional position coordinates of the cone barrels of each color relative to the monocular camera, and determining the distance of the cone barrels of each color relative to the driverless formula car based on the three-dimensional position coordinates; Detect whether the distance is less than a preset collision distance. If so, instruct the driverless formula car to adjust its driving direction based on cone barrels of different colors to avoid a collision between the driverless formula car and the driverless formula car.

5. The obstacle detection method for an unmanned formula racing car according to any one of claims 1 to 4, characterized in that: The basic network architecture of the cone bucket detection model is the YoloV7 model.

6. An obstacle detection device for an unmanned formula racing car, characterized in that: include: a track image acquisition module, configured to respond to an obstacle detection instruction of the unmanned formula car and acquire a track scene image from a monocular camera of the unmanned formula car; a cone barrel two-dimensional frame determination module, configured to perform target detection on the track scene image based on a pre-trained cone barrel detection model, and determine the two-dimensional frames of cone barrels of various colors in the track scene image; An overdetermined equation group determination module is configured to assume a three-dimensional frame of the cone barrel based on the two-dimensional frame of the cone barrel, construct a projection combination associated with the three-dimensional frame, and construct an overdetermined equation group associating pixel coordinates and the three-dimensional coordinates of the cone barrel based on the projection point pairs in the projection combination, which includes: Taking the center point of the three-dimensional frame of the cone barrel in the three-dimensional space as the origin of the world coordinate system, and determining the eight corner points of the three-dimensional frame of the cone barrel in the world coordinate system; Projecting the eight corner points of the three-dimensional frame of the cone barrel in the world coordinate system into the camera coordinate system to construct a projection combination of the three-dimensional frame and the two-dimensional frame, and constructing an overdetermined set of equations relating pixel coordinates and the three-dimensional coordinates of the cone barrel based on the projection point pairs in the projection combination; The three-dimensional coordinate determination module is configured to solve the overdetermined equations using the least squares method to determine the three-dimensional position coordinates of the cone barrel in the camera coordinate system, and includes: Solving the overdetermined system of equations based on multiple projection combinations of the three-dimensional box and the two-dimensional box using a least squares method; The center of the cone barrel's 3D frame obtained by each projection combination is projected onto the 2D image to determine the projection point, the projection point is compared with the center point of the cone barrel's 2D frame, and the solution corresponding to the projection combination with the smallest pixel distance between the two points is selected as the 3D position coordinate of the cone barrel; The obstacle detection module is configured to determine the distance of the cone barrels of each color relative to the unmanned formula racing car based on the three-dimensional position coordinates, and instruct the unmanned formula racing car to adjust its driving direction according to the distance to complete the detection of obstacles of the unmanned formula racing car.

7. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images