Vehicle motion trail tracking method and device based on optical flow and electronic equipment

Abstract

The invention provides a vehicle motion trail tracking method and device based on optical flow and electronic equipment, and the method comprises the steps: firstly obtaining continuous multiple frames of images of a target vehicle in a preset time period, then carrying out the iteration prediction of the optical flow of each frame of image and the previous frame of image in the multiple frames of images through a preset optical flow model, obtaining an optical flow graph corresponding to each frame of image, and carrying out the tracking of the motion trail of the target vehicle. Then determining a preliminary translation distance and a preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow diagram corresponding to each frame of image; and finally, based on the initial translation distance and the initial rotation angle of the target vehicle corresponding to each frame of image and a preset template corresponding to each frame of image, generating a motion track of the target vehicle in a preset time period. By adopting the method and the device, the positioning efficiency and accuracy can be improved, and related manual operation required by positioning is simplified at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle motion trajectory tracking, and in particular to a vehicle motion trajectory tracking method, device and electronic device based on optical flow. Background Art

[0002] An AGV (Automated Guided Vehicle) system, usually also referred to as an AGV cart, refers to a transport vehicle equipped with an automatic guiding device such as electromagnetic or optical, capable of traveling along a specified path and having safety protection and various on-vehicle functions. In industrial applications, the driverless AGV cart uses a rechargeable battery as its power source. Generally, the travel route and behavior of the AGV cart can be controlled by a computer, or an electromagnetic track can be set on the floor as the travel route of the AGV cart, so that the AGV cart moves relying on the electromagnetic track.

[0003] With the increasing automation of modern industrial production and logistics handling, visual navigation technology has important practical application value for path recognition and tracking control of AGV carts. For indoor positioning and navigation, traditional positioning methods mainly rely on the installation of magnetic strips, magnetic tapes, magnetic nails, etc. Although this positioning method has advantages such as controllable movement path and high safety, its path limitations are relatively large. On this basis, a slightly more flexible positioning method can use artificially preset laser reflectors, two-dimensional codes, ceilings, etc. as reference features to achieve positioning. A more flexible positioning method can adopt simultaneous localization and mapping (SLAM) technology, such as laser SLAM, etc. This positioning method mainly obtains the current position by matching the environmental features obtained by the current sensor with the pre-constructed environmental map, omitting the processes of manually setting road signs and searching for road signs. In addition, there is also a positioning method using WIFI to achieve positioning. Although the positioning accuracy of this positioning method is not very high, it has low environmental requirements and relatively simple positioning algorithms, and usually needs to be combined with other positioning methods for positioning. Compared with the above positioning methods, using visual navigation technology for positioning has simple operation and high real-time performance.

[0004] However, the positioning effect based on visual navigation technology often depends on the complexity of the environment. Once the environment becomes complex, the positioning effect is quite unsatisfactory, and a large amount of on-site data needs to be pre-annotated and processed. If an AGV cart is replaced or the scene is changed, a large amount of data needs to be re-annotated. In addition, factors such as the complex calculation and operation of the algorithm used for positioning based on visual navigation technology, dependence on prior target modeling, and the need for manual adjustment of some parameters make it difficult to popularize and apply this positioning method in actual applications. Summary of the Invention

[0005] In view of this, an object of the present invention is to provide a method, device and electronic device for tracking the motion trajectory of a vehicle based on optical flow, so as to alleviate the above problems existing in the related art.

[0006] In a first aspect, an embodiment of the present invention provides a method for tracking the motion trajectory of a vehicle based on optical flow. The method includes: obtaining a plurality of consecutive frames of images of a target vehicle within a preset time period; wherein the target vehicle is an AGV cart; iteratively predicting the optical flow between each frame of the plurality of frames of images and its previous frame through a preset optical flow model to obtain an optical flow map corresponding to each frame of image; wherein the end condition of the iterative prediction includes that the deviation between the last two iterative prediction results is less than a preset deviation threshold; determining a preliminary translation distance and a preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow map corresponding to each frame of image; wherein the preliminary rotation angle is determined based on the corresponding optical flow map and its corresponding preliminary translation distance; generating a motion trajectory of the target vehicle within the preset time period based on the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to each frame of image and a preset template corresponding to each frame of image; wherein the preset template includes a preset rectangular range and a preset rotation angle range.

[0007] In a second aspect, an embodiment of the present invention further provides a device for tracking the motion trajectory of a vehicle based on optical flow. The device includes: an obtaining module, configured to obtain a plurality of consecutive frames of images of a target vehicle within a preset time period; wherein the target vehicle is an AGV cart; a predicting module, configured to iteratively predict the optical flow between each frame of the plurality of frames of images and its previous frame through a preset optical flow model to obtain an optical flow map corresponding to each frame of image; wherein the end condition of the iterative prediction includes that the deviation between the last two iterative prediction results is less than a preset deviation threshold; a determining module, configured to determine a preliminary translation distance and a preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow map corresponding to each frame of image; wherein the preliminary rotation angle is determined based on the corresponding optical flow map and its corresponding preliminary translation distance; a generating module, configured to generate a motion trajectory of the target vehicle within the preset time period based on the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to each frame of image and a preset template corresponding to each frame of image; wherein the preset template includes a preset rectangular range and a preset rotation angle range.

[0008] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method for tracking the motion trajectory of a vehicle based on optical flow described in the first aspect above.

[0009] A method, device and electronic device for tracking a vehicle motion trajectory based on optical flow provided by an embodiment of the present invention first obtain multiple consecutive frames of images of a target vehicle within a preset time period, then iteratively predict the optical flow between each frame of the multiple frames of images and its previous frame through a preset optical flow model to obtain an optical flow map corresponding to each frame of image, and then determine the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow map corresponding to each frame of image. Finally, based on the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image and a preset template corresponding to each frame of image, the motion trajectory of the target vehicle within the preset time period is generated. By adopting the above technology, the motion trajectory of an AGV trolley can be detected in real time, the positioning efficiency and accuracy are improved compared with the existing AGV trolley positioning method, and at the same time, the related manual operations required for positioning are simplified, which is convenient for relevant personnel to timely master the traveling route of the AGV trolley and perform relevant processing in time.

[0010] Other features and advantages of the present invention will be described in the following specification, and part of them will become obvious from the specification, or be understood by implementing the present invention. The objectives and other advantages of the present invention are achieved and obtained by the structures specifically pointed out in the specification, claims and drawings.

[0011] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. Description of the Drawings

[0012] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 It is a schematic flowchart of a method for tracking a vehicle motion trajectory based on optical flow in an embodiment of the present invention;

[0014] Figure 2 It is an example flowchart of a method for tracking a vehicle motion trajectory based on optical flow in an embodiment of the present invention;

[0015] Figure 3 It is a schematic structural diagram of a device for tracking a vehicle motion trajectory based on optical flow in an embodiment of the present invention;

[0016] Figure 4 It is a schematic structural diagram of an electronic device in an embodiment of the present invention. Detailed Embodiments

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] Currently, for indoor positioning and navigation, traditional positioning methods mainly rely on the installation of magnetic strips, magnetic tapes, magnetic nails, etc. Although this positioning method has advantages such as controllable movement paths and high safety, the limitations of its paths are relatively large. On this basis, a slightly more flexible positioning method can use artificially preset laser reflectors, QR codes, ceilings, etc. as reference features to achieve positioning. A more flexible positioning method can adopt Simultaneous Localization and Mapping (SLAM) technology, such as laser SLAM, etc. This positioning method mainly obtains the current position by matching the environmental features obtained by the current sensor with the pre-constructed environmental map, omitting the processes of manually setting road signs and searching for road signs. In addition, there is also a positioning method that uses WIFI to achieve positioning. Although the positioning accuracy of this positioning method is not very high, it has low environmental requirements and relatively simple positioning algorithms, and usually needs to be combined with other positioning methods for positioning. Compared with the above positioning methods, using visual navigation technology for positioning has simple operations and high real-time performance.

[0019] However, the positioning effect based on visual navigation technology often depends on the complexity of the environment. Once the environment becomes complex, the positioning effect becomes quite unsatisfactory, and a large amount of on-site data needs to be pre-annotated and processed. If an AGV cart is replaced or the scene is changed, a large amount of data needs to be re-annotated. In addition, factors such as the complex calculation and operation of the algorithms used for positioning based on visual navigation technology, dependence on prior target modeling, and the need for manual adjustment of some parameters make it difficult to popularize and apply this positioning method in actual applications.

[0020] Based on this, an optical flow-based vehicle motion trajectory tracking method, device, and electronic device provided by the embodiments of the present invention can alleviate the above problems existing in the related technologies.

[0021] To facilitate the understanding of this embodiment, first, a detailed introduction to an optical flow-based vehicle motion trajectory tracking method disclosed in the embodiments of the present invention is provided. Refer to Figure 1 the flowchart of an optical flow-based vehicle motion trajectory tracking method shown in the figure. The method may include the following steps:

[0022] Step S102, obtain multiple consecutive frames of images of the target vehicle within a preset time period.

[0023] Among them, the target vehicle may be an AGV cart.

[0024] Step S104: Iteratively predict the optical flow between each frame of the multi-frame images and its previous frame through a preset optical flow model to obtain an optical flow map corresponding to each frame of image.

[0025] Among them, the end condition of the iterative prediction may include that the deviation between the results of the last two iterative predictions is less than a preset deviation threshold. For example, the preset optical flow model adopts the RAFT model, and the RAFT model performs a total of 12 iterations. That is to say, the RAFT model will generate 12 optical flow results at full resolution. The end condition of the iterative prediction of the RAFT model can be set as the deviation between the results of the last two iterative predictions being less than the preset deviation threshold, so that when the RAFT model meets the end condition, the iteration can be ended without necessarily completing all 12 iterations, thereby ending the iteration in advance without reducing the accuracy of optical flow prediction and improving the efficiency of optical flow prediction.

[0026] Step S106: Based on the optical flow maps corresponding to each frame of image, determine the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image.

[0027] Among them, the preliminary rotation angle is determined based on the corresponding optical flow map and its corresponding preliminary translation distance.

[0028] Since optical flow maps corresponding to each adjacent pair of frames are predicted, the translation distance of the target vehicle corresponding to the adjacent pair of frames can be calculated based on the optical flow results of all pixels in the area where the target vehicle is located in the optical flow maps corresponding to the adjacent pair of frames as the preliminary translation distance. Furthermore, the rotation angle of the target vehicle corresponding to the two frames can be calculated based on the optical flow results of all pixels in the area where the target vehicle is located in the optical flow maps corresponding to the adjacent pair of frames and the preliminary translation distance as the preliminary rotation angle.

[0029] Step S108: Based on the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image and the preset template corresponding to each frame of image, generate the motion trajectory of the target vehicle within a preset time period.

[0030] Among them, the preset template may include a preset rectangular range and a preset rotation angle range.

[0031] For every two adjacent frames of images, a corresponding preset rectangular range and a preset rotation angle range can be pre-configured for the two adjacent frames of images, and the preset rectangular range and the preset rotation angle range are combined to form a preset template as the first positioning result of the target vehicle corresponding to the two adjacent frames of images. After obtaining the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to the two adjacent frames of images as the second positioning result of the target vehicle corresponding to the two adjacent frames of images, the final positioning result of the target vehicle corresponding to the two adjacent frames of images can be determined by combining the first positioning result and the second positioning result; and so on, the final positioning results of the target vehicle corresponding to each frame of images can be obtained, and then the motion trajectory of the target vehicle within a preset time period can be generated by using this part of the obtained final positioning results. Through the above operation method of combining the preliminary translation distance and the preliminary rotation angle determined by the optical flow map with the preset rectangular range and the preset rotation angle range of the preset template, the accuracy of the generation of the motion trajectory of the target vehicle can be ensured.

[0032] A method for tracking a vehicle motion trajectory based on optical flow provided by an embodiment of the present invention first obtains a plurality of consecutive frames of images of a target vehicle within a preset time period, then iteratively predicts the optical flow between each frame of image and its previous frame of image in the plurality of frames of images through a preset optical flow model to obtain an optical flow map corresponding to each frame of image, and then determines the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow map corresponding to each frame of image, and finally generates the motion trajectory of the target vehicle within a preset time period based on the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to each frame of image and the preset template corresponding to each frame of image. By adopting the above technology, the motion trajectory of the AGV trolley can be detected in real time, the positioning efficiency and accuracy are improved compared with the existing AGV trolley positioning method, and at the same time, the relevant manual operations required for positioning are simplified, which is convenient for relevant personnel to timely master the traveling route of the AGV trolley and perform relevant processing in time.

[0033] As a possible implementation manner, the above method for tracking a vehicle motion trajectory based on optical flow may further include: detecting the target vehicle in each frame of image through a pre-trained image detection model to obtain a rectangular detection frame corresponding to each frame of image containing the target vehicle; determining the preset rectangular range and the preset rotation angle range corresponding to each frame of image based on the detection frames corresponding to each frame of image; and combining the preset rectangular frame and the preset rotation angle range corresponding to each frame of image to form a corresponding preset template.

[0034] Exemplarily, all consecutive multiple frames of images of the target vehicle within a preset time period can be input into a pre-trained image detection model in advance for detecting the target vehicle, so as to output a rectangular detection frame containing the target vehicle corresponding to each frame of image through the image detection model; then, the union of the coverage ranges of the detection frames corresponding to every two adjacent frames of images is taken as the corresponding preset rectangular range, and the maximum value (corresponding to the position of the detection frame with the largest offset) and the minimum value (corresponding to the position of the detection frame with the smallest offset) of the rotation angles corresponding to every two adjacent frames of images are calculated according to the offset of the center position and each corner point position of the detection frame corresponding to the current frame of image relative to the center position and each corner point position of the detection frame corresponding to the previous frame of image, and further, the range between the maximum value and the minimum value of the rotation angles corresponding to every two adjacent frames of images is taken as the corresponding preset rotation angle range; then, the preset rectangular range and the preset rotation angle range corresponding to every two adjacent frames of images are combined to form the corresponding preset template.

[0035] As a possible implementation manner, the above step S108 (that is, generating the motion trajectory of the target vehicle within a preset time period based on the preliminary translation distance and preliminary rotation angle corresponding to the target vehicle in each frame of image and the preset template corresponding to each frame of image) may include:

[0036] Step 1, determining the optimized translation distance and optimized rotation angle corresponding to the target vehicle in each frame of image based on the preliminary translation distance and preliminary rotation angle corresponding to the target vehicle in each frame of image and the preset template corresponding to each frame of image.

[0037] Exemplarily, in the above step 1, the preliminary translation distance corresponding to each frame of image can be matched with the corresponding preset rectangular range, and the corresponding optimized translation distance is obtained after the matching; the preliminary rotation angle corresponding to each frame of image can be matched with the corresponding preset rotation angle range, and the corresponding optimized rotation angle is obtained after the matching.

[0038] Step 2, generating the motion trajectory of the target vehicle within a preset time period based on the optimized translation distance and optimized rotation angle corresponding to the target vehicle in each frame of image.

[0039] Through the above operation method of matching the preliminary translation distance and preliminary rotation angle determined by the optical flow map with the preset rectangular range and preset rotation angle range of the preset template in step 1 and step 2 above, the further optimization of the translation distance and rotation angle can be respectively realized, and further, the accuracy of the subsequent generation of the motion trajectory of the target vehicle can be ensured.

[0040] As a possible implementation, each pixel in the above optical flow map has a corresponding optical flow vector composed of a vertical component and a horizontal component; based on this, the above step S106 (that is, based on the optical flow maps corresponding to each frame of image, determining the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image) may include:

[0041] Step A, calculate the mean value of all optical flow vectors of the optical flow map corresponding to each frame of image, and determine the corresponding preliminary translation distance based on the mean value calculated for each frame of image.

[0042] Exemplarily, since each pixel in the area of the target vehicle in the optical flow maps corresponding to every two adjacent frames of images has a corresponding optical flow vector composed of a vertical component and a horizontal component, and the vertical component and horizontal component of each pixel are respectively used to represent the translation distance of the pixel in the vertical direction and the horizontal direction, and the optical flow vectors of different pixels are usually different. Therefore, in order to represent the translation distance of the target vehicle in the vertical direction and the horizontal direction, the mean value of the optical flow vectors of all pixels in the area of the target vehicle in the optical flow maps corresponding to every two adjacent frames of images can be calculated as the preliminary translation distance of the target vehicle corresponding to these two adjacent frames of images. By analogy, the preliminary translation distances of the target vehicle corresponding to each frame of image can be obtained.

[0043] Step B, based on the optical flow map and the preliminary translation distance corresponding to each frame of image, determine the preliminary rotation angle corresponding to each frame of image.

[0044] Exemplarily, the above step B can be carried out according to the following operation method:

[0045] Step B1, for each frame of image, subtract each optical flow vector of the optical flow map corresponding to this frame of image from the corresponding preliminary translation distance to obtain the optimized optical flow map corresponding to this frame of image.

[0046] Exemplarily, for the i-th pixel in a certain optical flow map, the optical flow vector of this pixel can be denoted as I i =(u i , v i ), u i and v i are respectively the vertical component and horizontal component of I i . Correspondingly, the initial translation distance corresponding to this optical flow map can be denoted as I=(u, v), where u and v are respectively the vertical component and horizontal component of I. Through the formula I i ’=(u i -u, v i -v), the optical flow vector I i’ and so on. The optical flow vectors of all pixels in the optimized optical flow map corresponding to each optical flow map can be calculated in sequence, so as to obtain the optimized optical flow map corresponding to each optical flow map.

[0047] Step B2: Based on the optical flow map and the optimized optical flow map corresponding to each frame of image, determine the preliminary rotation angle corresponding to each frame of image.

[0048] Exemplarily, in the above step B2, the preliminary rotation center corresponding to each frame of image can be determined based on the optimized optical flow map corresponding to each frame of image; then, based on the optical flow map and the preliminary rotation center corresponding to each frame of image, a corresponding rotation matrix is generated for each frame of image; then, based on the optical flow map and the rotation matrix corresponding to each frame of image, the preliminary rotation angle corresponding to each frame of image is determined.

[0049] Continuing with the previous example, after obtaining the optimized optical flow maps corresponding to each optical flow map, the maximum between-class variance method (i.e., the Otsu algorithm, also known as the Otsu method) can be used to determine, from the pixels within the region of the target vehicle in each optimized optical flow map, the pixel with the largest difference in optical flow vector from the optical flow vectors of all pixels within the region of non-target vehicles as the corresponding preliminary rotation center; then, based on the optical flow vectors of the pixels within the region of the target vehicle in each optical flow map, the rotation angles of the positions of the pixels within the region of the target vehicle in each optical flow map (such as the corner positions of the region of the target vehicle) relative to the pixel position corresponding to the preliminary rotation center are calculated, and then, based on the rotation angles of the positions of the pixels within the region of the target vehicle in each optical flow map, a corresponding rotation matrix is generated for each optical flow map, where each element in the rotation matrix is the rotation angle of the corresponding pixel position within the region of the target vehicle in the optical flow map; then, the mean value of all elements in each rotation matrix is calculated, and for each rotation matrix, the least squares method is used to calculate the corresponding value that minimizes the sum of variances of all elements in each rotation matrix as the corresponding preliminary rotation angle, so as to obtain the preliminary rotation angles corresponding to each adjacent two frames of images.

[0050] As a possible implementation manner, the above vehicle motion trajectory tracking method based on optical flow may further include: determining whether the motion trajectory matches a preset motion trajectory; if the motion trajectory does not match the preset motion trajectory, an alarm is given. By adopting this operation mode, an alarm can be given in a timely manner when the motion trajectory of the target vehicle deviates from the preset motion trajectory, so that relevant personnel can handle it immediately after receiving the alarm.

[0051] For ease of understanding, an exemplary description of the operation mode of the above vehicle motion trajectory tracking method based on optical flow is given below by taking a specific application as an example.

[0052] All the steps of the above vehicle motion trajectory tracking method based on optical flow can be written into an executable algorithm in advance. See Figure 2As shown below, the overall process of the algorithm is as follows:

[0053] First step, initialize the RAFT model.

[0054] Since the RAFT model has a slow speed in performing optical flow detection on the first frame image of the AGV vehicle within a preset time period and several consecutive frames of images after it, before starting the optical flow detection of the RAFT model, it is necessary to input the first frame image of the AGV vehicle within a preset time period and several consecutive frames of images after it into the RAFT model for one or more optical flow detections to warm up the RAFT model, so as to ensure that the RAFT model has a fast optical flow detection speed after starting the optical flow detection.

[0055] Second step, perform optical flow detection on the RAFT model.

[0056] The end condition for the optical flow detection of the RAFT model can be preset as the deviation between the optical flow detection results of the last two iterations being less than a preset deviation threshold. Then, when inputting multiple consecutive frames of images of the AGV vehicle within a preset time period into the RAFT model for optical flow detection, the RAFT model can end the optical flow detection when the end condition is met and output the optical flow detection result obtained in the last iteration.

[0057] Third step, calculate the movement direction and movement position.

[0058] The method adopted is a combination of direct optical flow calculation and template matching: first, calculate the general movement position and movement direction (i.e., rotation angle) of the AGV vehicle through the optical flow detection result data of the AGV vehicle, and then update the accurate movement position and movement direction of the AGV vehicle through template matching, so as to prevent the accumulation of positioning errors in multiple consecutive frames of images.

[0059] Fourth step, perform Kalman filtering on the movement position.

[0060] Since the movement position information obtained by positioning the AGV vehicle in multiple consecutive frames of images includes the influence of noise and interference, the movement position information can be filtered by performing Kalman filtering on the movement position information to filter out the noise and interference in the movement position information, thereby realizing the optimization of the movement position information; then, based on the optimized movement position information of the AGV vehicle in multiple consecutive frames of images, the movement trajectory of the AGV vehicle within a preset time period can be generated. By performing Kalman filtering on the movement position, the generated movement trajectory can be made more accurate and smooth.

[0061] In the above vehicle motion trajectory tracking method based on optical flow, an optical flow model is used to track the motion trajectory of the AGV vehicle. Since it does not require a large amount of labeled data compared to other traditional object detection algorithms (such as Yolo, etc.), the applicable scenario range of this method is larger. At the same time, the motion position and motion direction of the AGV vehicle can be calculated more directly and quickly according to the optical flow detection results of the optical flow model, which can meet the relevant application requirements for the AGV vehicle to track the motion trajectory in an increasingly complex site environment.

[0062] Based on the above vehicle motion trajectory tracking method based on optical flow, an embodiment of the present invention further provides a vehicle motion trajectory tracking device based on optical flow. Refer to Figure 3 As shown, the device may include the following modules:

[0063] An acquisition module 302, configured to acquire a plurality of consecutive frames of images of a target vehicle within a preset time period; wherein, the target vehicle is an AGV vehicle.

[0064] A prediction module 304, configured to iteratively predict the optical flow between each frame of the plurality of frames of images and its previous frame through a preset optical flow model to obtain an optical flow map corresponding to each frame of image; wherein, the end condition of the iterative prediction includes that the deviation between the last two iterative prediction results is less than a preset deviation threshold.

[0065] A determination module 306, configured to determine a preliminary translation distance and a preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow map corresponding to each frame of image; wherein, the preliminary rotation angle is determined based on the corresponding optical flow map and its corresponding preliminary translation distance.

[0066] A generation module 308, configured to generate a motion trajectory of the target vehicle within a preset time period based on the preliminary translation distance of the target vehicle corresponding to each frame of image, the preliminary rotation angle, and a preset template corresponding to each frame of image; wherein, the preset template includes a preset rectangular range and a preset rotation angle range.

[0067] A vehicle motion trajectory tracking device based on optical flow provided by an embodiment of the present invention first obtains a plurality of consecutive frames of images of a target vehicle within a preset time period, then iteratively predicts the optical flow between each frame of image and its previous frame of image in the plurality of frames of images through a preset optical flow model to obtain an optical flow map corresponding to each frame of image, and then determines a preliminary translation distance and a preliminary rotation angle of the target vehicle corresponding to each frame of image based on the optical flow maps corresponding to each frame of image. Finally, based on the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to each frame of image and a preset template corresponding to each frame of image, a motion trajectory of the target vehicle within the preset time period is generated. By adopting the above technology, the motion trajectory of the AGV cart can be detected in real time, the positioning efficiency and accuracy are improved compared with the existing AGV cart positioning method, and at the same time, the relevant manual operations required for positioning are simplified, which is convenient for relevant personnel to timely master the traveling route of the AGV cart and perform relevant processing in time.

[0068] The above-mentioned generation module 308 can also be used to: determine an optimized translation distance and an optimized rotation angle of the target vehicle corresponding to each frame of image based on the preliminary translation distance and the preliminary rotation angle of the target vehicle corresponding to each frame of image and a preset template corresponding to each frame of image; generate a motion trajectory of the target vehicle within the preset time period based on the optimized translation distance and the optimized rotation angle of the target vehicle corresponding to each frame of image.

[0069] Each pixel in the above-mentioned optical flow map has a corresponding optical flow vector composed of a vertical component and a horizontal component; based on this, the above-mentioned determination module 306 can also be used to: calculate the mean value of all the optical flow vectors of the optical flow map corresponding to each frame of image, and determine the corresponding preliminary translation distance based on the mean value calculated for each frame of image; determine the preliminary rotation angle corresponding to each frame of image based on the optical flow map and the preliminary translation distance corresponding to each frame of image.

[0070] The above-mentioned determination module 306 can also be used to: for each frame of image, subtract each optical flow vector of the optical flow map corresponding to this frame of image from the corresponding preliminary translation distance to obtain an optimized optical flow map corresponding to this frame of image; determine the preliminary rotation angle corresponding to each frame of image based on the optical flow map and the optimized optical flow map corresponding to each frame of image.

[0071] The above-mentioned determination module 306 can also be used to: determine the preliminary rotation center corresponding to each frame of image based on the optimized optical flow map corresponding to each frame of image; generate a corresponding rotation matrix for each frame of image based on the optical flow map and the preliminary rotation center corresponding to each frame of image; determine the preliminary rotation angle corresponding to each frame of image based on the optical flow map and the rotation matrix corresponding to each frame of image.

[0072] The above-mentioned generation module 308 can also be used to: match the preliminary translation distance corresponding to each frame of image with the corresponding preset rectangular range, and obtain the corresponding optimized translation distance after matching; match the preliminary rotation angle corresponding to each frame of image with the corresponding preset rotation angle range, and obtain the corresponding optimized rotation angle after matching.

[0073] See Figure 3 As shown, the device may further include:

[0074] A template establishment module 310, configured to detect the target vehicle in each frame of image through a pre-trained image detection model, and obtain a rectangular detection frame including the target vehicle corresponding to each frame of image; determine the corresponding preset rectangular range and preset rotation angle range for each frame of image based on the detection frames corresponding to each frame of image; and form the corresponding preset template by combining the preset rectangular frame and preset rotation angle range corresponding to each frame of image.

[0075] A judgment module 312, configured to judge whether the motion trajectory matches a preset motion trajectory.

[0076] An alarm module 314, configured to give an alarm when the motion trajectory does not match the preset motion trajectory.

[0077] The device for tracking a vehicle motion trajectory based on optical flow provided by an embodiment of the present invention has the same implementation principle and the same technical effects as those of the foregoing method embodiment for tracking a vehicle motion trajectory based on optical flow. For a brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment.

[0078] An embodiment of the present invention further provides an electronic device, as Figure 4 shown, which is a schematic structural diagram of the electronic device. Among them, the electronic device includes a processor 41 and a memory 40. The memory 40 stores computer-executable instructions that can be executed by the processor 41, and the processor 41 executes the computer-executable instructions to implement the foregoing method for tracking a vehicle motion trajectory based on optical flow.

[0079] In Figure 4 the shown embodiment, the electronic device further includes a bus 42 and a communication interface 43. Among them, the processor 41, the communication interface 43, and the memory 40 are connected through the bus 42.

[0080] Among them, the memory 40 may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk memory. The communication connection between this system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 42 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a bidirectional arrow is used in Figure 4 , but it does not mean that there is only one bus or one type of bus.

[0081] The processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 41 or instructions in software form. The above-mentioned processor 41 can be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it can also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field programmable gate array (FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor 41 reads the information in the memory and combines its hardware to complete the steps of the method for tracking the vehicle motion trajectory based on optical flow in the foregoing embodiments.

[0082] Unless otherwise specifically stated, the relative steps, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0083] If the described functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0084] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0085] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments or easily conceive of changes, or perform equivalent replacements for some of the technical features; and these modifications, changes, or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for tracking the motion trajectory of a vehicle based on optical flow, characterized in that, the method includes: Obtain multiple consecutive frames of images of the target vehicle within a preset time period; wherein, the target vehicle is an AGV cart; Iteratively predict the optical flow between each frame of the multiple frames of images and its previous frame through a preset optical flow model to obtain an optical flow map corresponding to each frame of image; wherein, the end condition of the iterative prediction includes that the deviation between the results of the last two iterative predictions is less than a preset deviation threshold; Based on the optical flow maps corresponding to each frame of image, determine the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image; wherein, the preliminary rotation angle is determined based on the corresponding optical flow map and its corresponding preliminary translation distance; Based on the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image and the preset template corresponding to each frame of image, generate the motion trajectory of the target vehicle within the preset time period; wherein, the preset template includes a preset rectangular range and a preset rotation angle range.

2. The method according to claim 1, characterized in that, Generating the motion trajectory of the target vehicle within the preset time period based on the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image and the preset template corresponding to each frame of image includes: Based on the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image and the preset template corresponding to each frame of image, determine the optimized translation distance and optimized rotation angle of the target vehicle corresponding to each frame of image; Based on the optimized translation distance and optimized rotation angle of the target vehicle corresponding to each frame of image, generate the motion trajectory of the target vehicle within the preset time period.

3. The method according to claim 1, characterized in that, Each pixel in the optical flow map has a corresponding optical flow vector composed of a vertical component and a horizontal component; Based on the optical flow maps corresponding to each frame of image, determining the preliminary translation distance and preliminary rotation angle of the target vehicle corresponding to each frame of image includes: Calculate the mean value of all optical flow vectors of the optical flow map corresponding to each frame of image, and determine the corresponding preliminary translation distance based on the mean value calculated for each frame of image; Based on the optical flow map and preliminary translation distance corresponding to each frame of image, determine the preliminary rotation angle corresponding to each frame of image.

4. The method according to claim 3, characterized in that, Based on the optical flow map and preliminary translation distance corresponding to each frame of image, determining the preliminary rotation angle corresponding to each frame of image includes: For each frame of image, subtract the corresponding preliminary translation distance from each optical flow vector of the optical flow map corresponding to this frame of image to obtain an optimized optical flow map corresponding to this frame of image; Based on the optical flow map and optimized optical flow map corresponding to each frame of image, determine the preliminary rotation angle corresponding to each frame of image.

5. The method according to claim 4, characterized in that, Based on the optical flow map and optimized optical flow map corresponding to each frame of image, determining the preliminary rotation angle corresponding to each frame of image includes: Based on the optimized optical flow map corresponding to each frame of image, determine the preliminary rotation center corresponding to each frame of image; Generate a corresponding rotation matrix for each frame of image based on the optical flow map and the preliminary rotation center corresponding to each frame of image; Determine the preliminary rotation angle corresponding to each frame of image based on the optical flow map and the rotation matrix corresponding to each frame of image.

6. The method according to claim 2, wherein, Based on the preliminary translation distance, the preliminary rotation angle corresponding to each frame of image of the target vehicle, and the preset templates corresponding to each frame of image, determine the optimized translation distance and the optimized rotation angle corresponding to each frame of image of the target vehicle, including: Match the preliminary translation distance corresponding to each frame of image with the corresponding preset rectangular range, and obtain the corresponding optimized translation distance after matching; Match the preliminary rotation angle corresponding to each frame of image with the corresponding preset rotation angle range, and obtain the corresponding optimized rotation angle after matching.

7. The method according to claim 1, wherein, The method further includes: Detect the target vehicle in each frame of image through a pre-trained image detection model, and obtain a rectangular detection frame containing the target vehicle corresponding to each frame of image; Based on the detection frames corresponding to each frame of image, determine the preset rectangular range and the preset rotation angle range corresponding to each frame of image; Form the corresponding preset template by combining the preset rectangular frame and the preset rotation angle range corresponding to each frame of image.

8. The method according to claim 1, wherein, The method further includes: Judge whether the motion trajectory matches a preset motion trajectory; If the motion trajectory does not match the preset motion trajectory, an alarm is issued.

9. An optical flow-based vehicle motion trajectory tracking device, wherein, The device includes: An acquisition module, configured to acquire a plurality of consecutive frames of images of a target vehicle within a preset time period; wherein, the target vehicle is an AGV cart; A prediction module, configured to iteratively predict the optical flow between each frame of image and its previous frame of image in the plurality of frames of images through a preset optical flow model, and obtain an optical flow map corresponding to each frame of image; wherein, the end condition of the iterative prediction includes that the deviation between the last two iterative prediction results is less than a preset deviation threshold; A determination module, configured to determine the preliminary translation distance and the preliminary rotation angle corresponding to each frame of image of the target vehicle based on the optical flow maps corresponding to each frame of image; wherein, the preliminary rotation angle is determined based on the corresponding optical flow map and its corresponding preliminary translation distance; A generation module, configured to generate the motion trajectory of the target vehicle within a preset time period based on the preliminary translation distance, the preliminary rotation angle corresponding to each frame of image of the target vehicle, and the preset templates corresponding to each frame of image; wherein, the preset template includes a preset rectangular range and a preset rotation angle range.

10. An electronic device, wherein, It includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the optical flow-based vehicle motion trajectory tracking method according to any one of claims 1 to 8.

Images