Object tracking method and device based on neural network, equipment and storage medium

Abstract

The application discloses a kind of object tracking method, device and equipment based on neural network and storage medium, by current driving image;Utilize preset convolutional neural network model to carry out image recognition to current driving image, obtain the image recognition result of target object, then root the image recognition result whether target object is tracking object;If yes, target object is tracked.Due to being through the convolutional neural network model based on deep learning to identify driving image, then according to the identification result to determine tracking object, can guarantee the accuracy of object tracking.Meanwhile, since the neural network model used is the simplified neural network constituted by the first stage for carrying out image feature extraction and containing down-sampling convolution layer and the second stage for carrying out image category prediction, so that the embedded device running the neural network is not limited by the amount of calculation and memory bandwidth, guarantee the speed and accuracy requirement of image recognition algorithm.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to an object tracking method, apparatus, device, and storage medium based on neural networks. Background Technology

[0002] Convolutional Neural Networks (CNNs) are deep neural network algorithms composed of operators such as convolution. They are one of the representative algorithms of deep learning and are widely used in machine vision and image recognition. Image recognition based on CNNs requires significant computational resources and memory access. On embedded devices, computational resources and memory bandwidth are limited, and existing CNNs cannot meet the speed and accuracy requirements of image recognition algorithms on embedded devices.

[0003] In the field of autonomous driving, forward collision warning (FCL) technology is a type of advanced driver assistance system (ADAS). It uses sensors to determine the distance and relative speed between the vehicle and the vehicle in front, and warns the driver when a potential collision risk exists. FCL technology can effectively improve driving safety, reduce traffic accidents, and minimize economic losses.

[0004] In forward collision warning technology, target tracking is a crucial component. Existing object tracking technologies, limited by the computing power of embedded devices, cannot utilize deep learning-based tracking techniques, thus failing to meet practical object tracking requirements.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for object tracking based on neural networks, aiming to solve the technical problem that existing object tracking technologies are limited by the computing power of embedded devices, making it impossible to use deep learning-based tracking technologies for object tracking and thus failing to meet the actual needs of object tracking.

[0007] To achieve the above objectives, the present invention provides an object tracking method based on a neural network, the method comprising the following steps:

[0008] The image recognition method based on convolutional neural networks includes:

[0009] Acquire the current driving images captured by the dashcam;

[0010] The current driving image is image recognized by a preset convolutional neural network model to obtain the image recognition result of the target object. The preset convolutional neural network model includes a first stage for image feature extraction and a second stage for image category prediction. The first stage includes a convolutional layer with downsampling.

[0011] Based on the image recognition results, determine whether the target object is a tracking object;

[0012] When the target object is a tracking object, the target object is tracked.

[0013] Optionally, the convolutional layer with downsampling includes a single convolutional layer module, a regular residual module, and a bottleneck residual module;

[0014] The step of performing image recognition on the current driving image using a preset convolutional neural network model to obtain the image recognition result of the target object includes:

[0015] The image features in the current driving image are downsampled sequentially through the single convolutional layer module, the conventional residual module, and the bottleneck residual module to obtain the target image features;

[0016] The second stage involves predicting the image category based on the target image features to obtain the image recognition result of the target object.

[0017] Optionally, the second stage includes an average pooling layer, a fully connected layer, and a Softmax layer;

[0018] The step of performing image category prediction based on the target image features in the second stage to obtain the image recognition result of the target object includes:

[0019] The target image features are pooled using the average pooling layer to obtain the dimensionality-reduced image features.

[0020] The fully connected layer is used to perform vector transformation on the dimensionality-reduced image features to obtain the image feature vector;

[0021] The image recognition result of the target object is generated by the Softmax layer based on the image feature vector.

[0022] Optionally, before the step of acquiring the current driving image captured by the dashcam, the method further includes:

[0023] Obtain the initial convolutional neural network model to be trained, as well as the preset image training set;

[0024] The initial convolutional neural network model is pre-trained based on the preset image training set to obtain a pre-trained convolutional neural network model.

[0025] The pre-trained convolutional neural network model is used to initialize the convolutional neural network model for image recognition to obtain a preset convolutional neural network model.

[0026] Optionally, the step of initializing the convolutional neural network model for image recognition based on the pre-trained convolutional neural network model to obtain a preset convolutional neural network model includes:

[0027] Obtain the model parameter weights corresponding to the pre-trained convolutional neural network model;

[0028] The convolutional neural network model for image recognition is initialized according to the model parameter weights to obtain a preset convolutional neural network model.

[0029] Optionally, the step of determining whether the target object is a tracking object based on the image recognition result includes:

[0030] Obtain the object type information and object location information of the target object from the image recognition results;

[0031] Based on the object type information and the object location information, determine whether the target object is a tracking object.

[0032] Optionally, before the step of determining whether the target object is a tracking object based on the image recognition result, the method further includes:

[0033] Obtain the previous frame of the current driving image;

[0034] Detect whether the target object exists in the previous frame of the driving image;

[0035] When the target object exists, the step of determining whether the target object is a tracking object based on the image recognition result is performed.

[0036] Furthermore, to achieve the above objectives, the present invention also proposes an object tracking device based on a neural network, wherein the image recognition device based on a convolutional neural network includes:

[0037] The image acquisition module is used to acquire the current driving images captured by the dashcam;

[0038] The image recognition module is used to perform image recognition on the current driving image through a preset convolutional neural network model to obtain the image recognition result of the target object. The preset convolutional neural network model includes a first stage for image feature extraction and a second stage for image category prediction. The first stage includes a convolutional layer with downsampling.

[0039] The object determination module is used to determine whether the target object is a tracking object based on the image recognition result.

[0040] The object tracking module is used to track the target object when the target object is a tracking object.

[0041] Furthermore, to achieve the above objectives, the present invention also proposes a neural network-based object tracking device, the device comprising: a memory, a processor, and a neural network-based object tracking program stored in the memory and executable on the processor, the neural network-based object tracking program being configured to implement the steps of the neural network-based object tracking method described above.

[0042] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a neural network-based object tracking program, which, when executed by a processor, implements the steps of the neural network-based object tracking method described above.

[0043] This invention acquires current driving images captured by a dashcam; uses a preset convolutional neural network model to perform image recognition on the current driving images, obtains the image recognition result of the target object, and then determines whether the target object is a tracking object based on the image recognition result; if the target object is a tracking object, it is tracked. Because this invention uses a deep learning-based convolutional neural network model to recognize driving images and then determines the tracking object based on the recognition result, the accuracy of object tracking can be guaranteed. Furthermore, since the neural network model used in this embodiment is a simplified neural network consisting of a first stage for image feature extraction containing downsampling convolutional layers and a second stage for image category prediction, the embedded device running this neural network is not limited by computational load and memory bandwidth, ensuring the speed and accuracy requirements of the image recognition algorithm. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of a neural network-based object tracking device in the hardware operating environment involved in the embodiments of the present invention;

[0045] Figure 2 This is a flowchart illustrating the first embodiment of the neural network-based object tracking method of the present invention;

[0046] Figure 3 This is a flowchart illustrating the second embodiment of the neural network-based object tracking method of the present invention;

[0047] Figure 4 This is a flowchart illustrating the third embodiment of the neural network-based object tracking method of the present invention;

[0048] Figure 5 This is a schematic diagram of distance calculation in the third embodiment of the neural network-based object tracking method of the present invention;

[0049] Figure 6 This is a structural block diagram of the first embodiment of the neural network-based object tracking device of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0052] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a neural network-based object tracking device in the hardware operating environment of an embodiment of the present invention.

[0053] like Figure 1 As shown, the neural network-based object tracking device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0054] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on neural network-based object tracking devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0055] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a neural network-based object tracking program.

[0056] exist Figure 1 In the neural network-based object tracking device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the neural network-based object tracking device of the present invention can be set in the neural network-based object tracking device, and the neural network-based object tracking device calls the neural network-based object tracking program stored in the memory 1005 through the processor 1001 and executes the neural network-based object tracking method provided in the embodiment of the present invention.

[0057] This invention provides an object tracking method based on a neural network, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the neural network-based object tracking method of the present invention.

[0058] In this embodiment, the neural network-based object tracking method includes the following steps:

[0059] Step S10: Acquire the current driving image captured by the dashcam;

[0060] It should be noted that the executing entity of the method in this embodiment can be an in-vehicle terminal connected to a dashcam or other embedded device with data processing, network communication, and program execution functions, and this embedded device can be installed in a vehicle. The following description uses an embedded device as an example to illustrate a neural network-based object tracking method of the present invention.

[0061] It should be understood that the current driving image can be one frame or multiple frames captured by the dashcam at the current moment.

[0062] Step S20: Perform image recognition on the current driving image using a preset convolutional neural network model to obtain the image recognition result of the target object. The preset convolutional neural network model includes a first stage for image feature extraction and a second stage for image category prediction. The first stage includes a convolutional layer with downsampling.

[0063] It should be noted that the preset convolutional neural network model can be a pre-constructed simplified convolutional neural network. A simplified neural network is a neural network obtained by simplifying the structure of the network model while maintaining the original performance of the neural network. In this embodiment, the preset convolutional neural network model is a simplified neural network consisting of a first stage and a second stage. To ensure the accuracy of the simplified neural network, the first stage also includes a convolutional layer with downsampling. In this embodiment, the downsampling operation, which was originally performed by the pooling layer, is placed in the convolutional layer, and then only an average pooling layer is set in the second stage to perform the pooling operation, which enables the simplified neural network to have higher accuracy.

[0064] Furthermore, as one implementation, in this step, the convolutional layer with downsampling may include a single convolutional layer module, a regular residual module, and a bottleneck residual module; correspondingly, the embedded device may sequentially downsample the image features in the current driving image through the single convolutional layer module, the regular residual module, and the bottleneck residual module to obtain target image features; then, through the second stage, image category prediction is performed based on the target image features to obtain the image recognition result of the target object.

[0065] Furthermore, to ensure the accuracy of the simplified neural network model, the second stage in this step includes an average pooling layer, a fully connected layer, and a Softmax layer. Accordingly, the embedded device performs pooling operations on the target image features through the average pooling layer to obtain dimensionality-reduced image features; performs vector transformation on the dimensionality-reduced image features through the fully connected layer to obtain an image feature vector; and generates the image recognition result of the target object based on the image feature vector through the Softmax layer.

[0066] The model structure of the preset convolutional neural network model in this embodiment can be shown in Table 1 below.

[0067]

[0068]

[0069] Table 1. Model Structure of Preset Convolutional Neural Network Model

[0070] Referring to Table 1 above, the first stage in this embodiment includes a convolutional layer with downsampling. This convolutional layer with downsampling includes a single convolutional layer module (containing network structures with output spatial resolutions of 112*112 and 56*56 in Table 1, i.e., containing two single convolutional layers), a regular residual module (containing network structures with output spatial resolutions of 28*28 in Table 1, i.e. containing three residual building blocks, each residual building block consisting of two convolutional layers), and a bottleneck residual module (containing network structures with output spatial resolutions of 14*14 and 7*7 in Table 1, i.e. consisting of several bottleneck residual blocks, each bottleneck residual block consisting of convolutional layers with kernel sizes of 1×1, 3×3, and 1×1 in sequence).

[0071] Accordingly, referring to Table 1 above, the second stage in this embodiment may include an average pooling layer, a fully connected layer (1000-d FC, a fully connected layer with 1000 neurons) and a softmax layer.

[0072] In a practical implementation, the embedded device can use the simplified pre-defined convolutional neural network model to perform image recognition on the current driving image to obtain the image recognition result of the target object.

[0073] Step S30: Determine whether the target object is a tracking object based on the image recognition result;

[0074] It should be noted that the tracked object can be anything that may affect driving operations during vehicle operation, such as vehicles, pedestrians, road obstacles, flower beds, and traffic facilities within the vehicle's forward field of vision. In this embodiment, the tracked object can include static objects and / or dynamic objects. Static objects are those that are stationary at the current moment or when the vehicle passes by; dynamic objects are those that are in motion or moving.

[0075] To ensure vehicle driving safety, the tracking object in this embodiment is preferably a dynamic object. In this embodiment, the embedded device can determine whether the identified objects (i.e., the target objects, the number of which is not limited) in the image belong to the tracking object based on the image recognition results. For example, the target object can be determined as a tracking object based on the object type (dynamic object or static object) corresponding to the target object in the image recognition results.

[0076] Furthermore, the object type can be initially determined based on the object's state under normal circumstances. For example, flower beds and vegetation in flower beds, traffic lights, etc. can be defined as static objects; vehicles and pedestrians can be defined as quasi-dynamic objects first, and then further determined as to whether they belong to dynamic objects based on specific object characteristics (such as whether the vehicle's taillights are on, whether the pedestrian is in a moving posture, etc.).

[0077] Step S40: When the target object is a tracking object, track the target object.

[0078] In practical implementation, once the embedded device identifies the target object as the object to be tracked, it can track it in real time using images captured by the dashcam to prevent collisions.

[0079] Furthermore, to ensure that the aforementioned preset convolutional neural network model can accurately perform image recognition and improve recognition accuracy, this embodiment further includes the following step before step S10:

[0080] Step S01: Obtain the initial convolutional neural network model to be trained, and the preset image training set;

[0081] It should be noted that the initial convolutional neural network model mentioned in this step, i.e. the simplified convolutional neural network model that has not been trained, can be pre-trained using a pre-defined image training set in order to apply the simplified convolutional neural network model to the field of image recognition.

[0082] In this embodiment, the preset image training set can be selected from the commonly used pre-training dataset "ILSVRC2012 dataset" in image classification datasets. This dataset contains 1000 class image packs, and each class image pack contains a large number of images.

[0083] Step S02: Pre-train the initial convolutional neural network model according to the preset image training set to obtain a pre-trained convolutional neural network model;

[0084] It should be understood that after obtaining the aforementioned preset image training set, the initial convolutional neural network model can be pre-trained using the image training set to obtain a pre-trained convolutional neural network model.

[0085] It should be emphasized that, in this embodiment, the training loss function of the initial convolutional neural network model is the cross-entropy loss function.

[0086] Step S03: Initialize the convolutional neural network model for image recognition based on the pre-trained convolutional neural network model to obtain a preset convolutional neural network model.

[0087] It should be understood that when the model training is complete, the model can be validated and tested using pre-set validation and test sets, allowing the optimal convolutional neural network model to be selected based on the validation and test results.

[0088] It should be noted that the convolutional neural network model used for image recognition is the convolutional neural network model set according to the actual visual task scenario (such as driving image recognition in this embodiment). Similarly, the convolutional neural network model in this embodiment is also a simplified convolutional neural network model, except that the model parameters of this model need to be adjusted according to the pre-trained convolutional neural network model, i.e., the initialization mentioned above. For example, two models are compared to determine the same model parameters, and then the weights of the corresponding model parameters in the convolutional neural network model used for image recognition are adjusted according to the weights of the same model parameters in the pre-trained convolutional neural network model.

[0089] Specifically, the model parameter weights corresponding to the pre-trained convolutional neural network model can be obtained; then, the convolutional neural network model for image recognition can be initialized according to the model parameter weights to obtain a preset convolutional neural network model.

[0090] This embodiment acquires the current driving image captured by a dashcam; uses a preset convolutional neural network model to perform image recognition on the current driving image, obtains the image recognition result of the target object, and then determines whether the target object is a tracking object based on the image recognition result; if the target object is a tracking object, it is tracked. Since this embodiment uses a deep learning-based convolutional neural network model to recognize the driving image and then determines the tracking object based on the recognition result, the accuracy of object tracking can be guaranteed. Furthermore, since the neural network model used in this embodiment is a simplified neural network consisting of a first stage for image feature extraction containing downsampling convolutional layers and a second stage for image category prediction, the embedded device running this neural network is not limited by computational load and memory bandwidth, ensuring the speed and accuracy requirements of the image recognition algorithm.

[0091] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the neural network-based object tracking method of the present invention.

[0092] Based on the first embodiment described above, in this embodiment, step S30 includes:

[0093] Step S301: Obtain the object type information and object location information of the target object from the image recognition result;

[0094] It should be noted that the object type information can be category information based on the object's inherent attributes (such as vehicles, people, numbers, traffic facilities, etc.). These inherent attributes can also include the object's state during normal use / operation, such as being stationary, moving, or able to switch between both. For example, a vehicle can be in a moving or stationary state, while a flower bed or traffic light is stationary. The object location information refers to the area where the object is located in the driving image, that is, the object's position information within the field of view in front of the vehicle.

[0095] It should be understood that both the object type information and the object location information can be obtained from the output of a preset convolutional neural network model. Specifically, the object type information can be obtained from the image category probabilities output by the softmax layer.

[0096] Step S302: Determine whether the target object is a tracking object based on the object type information and the object location information.

[0097] In practical implementation, after acquiring the object type and location information of the target object, the embedded device can determine whether the target object belongs to the tracked object based on this information. For example, the embedded device may identify, through a preset convolutional neural network model, the target objects in the current driving image as pedestrians (object type: dynamic objects), traffic lights (object type: static objects), zebra crossings (object type: static objects), and several vehicles (object type: dynamic objects). Then, based on the location information of the target object (located directly in front, to the left, to the right, etc., within the monitoring field of view of the dashcam), it can determine whether the target object belongs to the tracked object.

[0098] Furthermore, considering that the location of the target object has a higher priority than its category in preventing vehicle collisions, in this embodiment, the embedded device can first determine whether the target object is located within a preset tracking area based on the object location information; then, when the target object is located within the preset tracking area, it determines whether the target object is a tracking object based on the object type information. For example, although a roadside flower bed is a static object, if a vehicle travels towards the flower bed, it also needs to be treated as a tracking object for collision avoidance preprocessing.

[0099] The preset tracking area can be an area where there is a risk of vehicle collision, which can be divided according to the actual situation. These areas can be divided according to the degree of danger, such as the front, left front, and right front of the monitoring field of the dashcam.

[0100] As one implementation method, to accurately determine whether a target object is located within a preset tracking area, in this embodiment, the embedded device can also determine the boundary coordinates of the target object based on the object position information; then determine the bounding box of the target object based on the boundary coordinates; and finally determine whether the target object is located within the preset tracking area based on the area where the bounding box is located in the current driving image. The boundary coordinates can be expressed in the form of [left, top, width, height], where width and height refer to the width and height of the area occupied by the target object in the driving image, and left and top are the pixel coordinates of the upper left vertex of the target object.

[0101] Furthermore, to improve the efficiency of tracking object identification, in this embodiment, the embedded device can also read object type identifiers from the object type information; match the object type identifiers according to a preset tracking object list, and determine whether the target object is a tracking object based on the matching result. The preset tracking object list stores object type identifiers corresponding to the tracking objects. The object type identifier can be a label corresponding to the object, such as pedestrian, vehicle, traffic facility, etc., and the preset tracking object list can be a pre-defined set of type identifiers corresponding to the tracking objects.

[0102] This embodiment obtains the object type information and object location information of the target object from the image recognition results; then, it determines whether the target object is a tracking object based on the object type information and object location information. This can accurately determine whether the target object needs to be tracked, providing a reliable way to effectively avoid vehicle collision accidents.

[0103] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the neural network-based object tracking method of the present invention.

[0104] Based on the above embodiments, in this embodiment, before step S30, the method further includes:

[0105] Step S1: Obtain the previous frame of the current driving image;

[0106] It should be noted that, in order to accurately determine whether a target object is a tracked object, in this embodiment, before determining whether a target object is a tracked object based on the image recognition results, the embedded device can first detect whether the target object was already present in the dashcam's monitoring field of view, or whether it suddenly appeared in the monitoring field of view (e.g., a vehicle in the left or right lane suddenly changes lanes, or a pedestrian suddenly crosses the road). In this case, the embedded device can obtain the previous frame of the current driving image and then combine the two frames for analysis and judgment.

[0107] Step S2: Detect whether the target object exists in the previous frame of the driving image;

[0108] It should be understood that, in the specific implementation of this step, the embedded device can also use the above-mentioned preset convolutional neural network model to perform image recognition on the previous frame of the driving image to determine whether a target image exists.

[0109] Step S3: When the target object exists, perform the step of determining whether the target object is a tracking object based on the image recognition result.

[0110] It should be understood that if the target object is detected in the previous frame of the driving image, the object tracking can continue; if the target object is not detected in the previous frame of the driving image, it indicates that the target object may have appeared suddenly. A suddenly appearing object is undoubtedly more dangerous for vehicle driving. At this time, the embedded device should perform collision avoidance operation or prompt.

[0111] Furthermore, considering that a persistent target object may not pose a threat to driving safety in some situations, the distance between the object and the vehicle during vehicle movement is crucial for collision prevention. Therefore, it is possible to determine whether further tracking of the target object is necessary by calculating the distance between the target object and the vehicle where the embedded device is located. Specifically, when the target object is present in the previous frame of the driving image, the embedded device can obtain the distance between itself and the target object; then, if the distance is less than a preset safety distance, it can execute the step of determining whether the target object is a tracking object based on the image recognition result.

[0112] As one implementation method, when an embedded device detects the presence of a target object in the previous frame of a driving image, it can calculate the distance to the target object by establishing a coordinate system. Specifically, when the target object is present, a coordinate system can be established using the camera corresponding to the dashcam; based on this coordinate system, the distance to the target object can be obtained using perspective transformation technology. In practical applications, a spatial rectangular coordinate system can be established with the camera corresponding to the dashcam as the origin, and then the distance to the target object can be obtained based on this spatial rectangular coordinate system using perspective transformation technology.

[0113] To accurately calculate the distance, as another implementation, the coordinate system described above may include a world coordinate system and a camera coordinate system. Then, based on the world coordinate system and the camera coordinate system, the distance between the target object and the target object is obtained through perspective transformation techniques.

[0114] This section uses the world coordinate system, camera coordinate system, and vehicle as the target object to illustrate the above distance calculation.

[0115] Reference Figure 5 , Figure 5 This is a schematic diagram of distance calculation in the third embodiment of the neural network-based object tracking method of the present invention.

[0116] like Figure 5 As shown, a camera coordinate system and a world coordinate system are established with the optical center of the camera as the origin. In the world coordinate system, the z-axis is horizontally forward and the y-axis is vertically downward. Considering that the angle of the dashcam's (front vehicle's) camera may not be horizontal, a pitch angle α is assumed to exist between the camera coordinate system and the world coordinate system. Based on perspective transformation technology, the camera intrinsic parameter matrix A can be set as:

[0117]

[0118] Where f is the camera focal length, and Cv and Cu are the principal point offsets.

[0119] During the distance calculation, a point [x, h, z] is selected at the bottom of the vehicle in front, and its image coordinates are:

[0120]

[0121] After normalizing the image coordinates, we obtain the normalized image coordinates v:

[0122]

[0123] Similarly, selecting a point [x, h + Δh, z] on the top of the vehicle, its normalized image coordinates v + Δv are:

[0124]

[0125] Furthermore, considering that camera installation typically involves calibration, α is usually small, and that the distance to the vehicle in front is generally large while the vehicle is in motion (i.e., z is much larger than h), therefore:

[0126] Δv≈(Δh×f) / z

[0127] Where Δv is the height of the vehicle's bounding box in the image, and Δh is the height of the vehicle in the world coordinate system, the final distance estimate is: z≈(Δh×f) / Δv.

[0128] This embodiment can accurately determine whether a target object needs to be tracked through the above method. On the other hand, when the target object needs to be tracked, it can also calculate the distance between the target object and the target object relatively accurately, thereby preventing vehicle collisions.

[0129] Furthermore, embodiments of the present invention also propose a storage medium storing a neural network-based object tracking program, which, when executed by a processor, implements the steps of the neural network-based object tracking method described above.

[0130] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the neural network-based object tracking device of the present invention.

[0131] like Figure 6 As shown, the neural network-based object tracking device proposed in this embodiment of the invention includes:

[0132] Image acquisition module 601 is used to acquire the current driving image captured by the dashcam;

[0133] The image recognition module 602 is used to perform image recognition on the current driving image through a preset convolutional neural network model to obtain the image recognition result of the target object. The preset convolutional neural network model includes a first stage for image feature extraction and a second stage for image category prediction. The first stage includes a convolutional layer with downsampling.

[0134] The object determination module 603 is used to determine whether the target object is a tracking object based on the image recognition result;

[0135] The object tracking module 604 is used to track the target object when the target object is a tracking object.

[0136] This embodiment acquires the current driving image captured by a dashcam; uses a preset convolutional neural network model to perform image recognition on the current driving image, obtains the image recognition result of the target object, and then determines whether the target object is a tracking object based on the image recognition result; if the target object is a tracking object, it is tracked. Since this embodiment uses a deep learning-based convolutional neural network model to recognize the driving image and then determines the tracking object based on the recognition result, the accuracy of object tracking can be guaranteed. Furthermore, since the neural network model used in this embodiment is a simplified neural network consisting of a first stage for image feature extraction containing downsampling convolutional layers and a second stage for image category prediction, the embedded device running this neural network is not limited by computational load and memory bandwidth, ensuring the speed and accuracy requirements of the image recognition algorithm.

[0137] Based on the first embodiment of the neural network-based object tracking device of the present invention described above, a second embodiment of the neural network-based object tracking device of the present invention is proposed.

[0138] In this embodiment, the convolutional layer with downsampling includes a single convolutional layer module, a regular residual module, and a bottleneck residual module;

[0139] The image recognition module 602 is further configured to downsample the image features in the current driving image sequentially through the single convolutional layer module, the conventional residual module, and the bottleneck residual module to obtain the target image features;

[0140] The image recognition module 602 is further configured to perform image category prediction based on the target image features in the second stage to obtain the image recognition result of the target object.

[0141] In one implementation, the second stage includes an average pooling layer, a fully connected layer, and a Softmax layer;

[0142] The image recognition module 602 is further configured to perform pooling operation on the target image features through the average pooling layer to obtain the dimension-reduced image features;

[0143] The image recognition module 602 is further configured to perform vector transformation on the dimension-reduced image features through the fully connected layer to obtain an image feature vector;

[0144] The image recognition module 602 is further configured to generate an image recognition result of the target object based on the image feature vector through the Softmax layer.

[0145] In one embodiment, the device further includes a model training module;

[0146] The model training module is used to obtain the initial convolutional neural network model to be trained, as well as the preset image training set;

[0147] The model training module is also used to pre-train the initial convolutional neural network model according to the preset image training set to obtain a pre-trained convolutional neural network model.

[0148] The model training module is further configured to initialize the convolutional neural network model for image recognition based on the pre-trained convolutional neural network model, so as to obtain a preset convolutional neural network model.

[0149] In one implementation, the model training module is further configured to obtain the model parameter weights corresponding to the pre-trained convolutional neural network model; and initialize the convolutional neural network model for image recognition according to the model parameter weights to obtain a preset convolutional neural network model.

[0150] In one implementation, the object determination module 603 is further configured to obtain object type information and object location information of the target object from the image recognition result; and determine whether the target object is a tracking object based on the object type information and the object location information.

[0151] In one implementation, the object determination module 603 is further configured to determine whether the target object is located in a preset tracking area based on the object location information; and when the target object is located in the preset tracking area, to determine whether the target object is a tracking object based on the object type information.

[0152] In one implementation, the object determination module 603 is further configured to determine the boundary coordinates of the target object based on the object location information; determine the bounding box of the target object based on the boundary coordinates; and determine whether the target object is located in a preset tracking area based on the area where the bounding box is located in the current driving image.

[0153] In one implementation, the object determination module 603 is further configured to read an object type identifier from the object type information; match the object type identifier according to a preset tracking object list; and determine whether the target object is a tracking object based on the matching result. The preset tracking object list stores the object type identifier corresponding to the tracking object.

[0154] In one implementation, the object determination module 603 is further configured to acquire the previous frame of the current driving image; detect whether the target object exists in the previous frame of the driving image; and when the target object exists, perform the operation of determining whether the target object is a tracking object based on the image recognition result.

[0155] In one implementation, the object determination module 603 is further configured to, when the target object exists, obtain the distance between itself and the target object; and when the distance is less than a preset safe distance, perform the operation of determining whether the target object is a tracking object based on the image recognition result.

[0156] In one implementation, the object determination module 603 is further configured to establish a coordinate system with the camera corresponding to the dashcam when the target object exists; and to obtain the distance between the target object and the dashcam using perspective transformation technology based on the coordinate system.

[0157] As one implementation, the object determination module 603 is also used to obtain the distance between itself and the target object based on the world coordinate system and the camera coordinate system through perspective transformation technology.

[0158] Other embodiments or specific implementations of the object tracking device based on neural networks of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0159] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0160] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0162] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A neural network-based object tracking method, characterized by, The neural network-based image recognition method includes: Acquire the current driving images captured by the dashcam; The current driving image is image recognized by a preset convolutional neural network model to obtain the image recognition result of the target object. The preset convolutional neural network model includes a first stage for image feature extraction and a second stage for image category prediction. The first stage includes a convolutional layer with downsampling. Based on the image recognition results, determine whether the target object is a tracking object; When the target object is a tracking object, the target object is tracked; The convolutional layer with downsampling includes a single convolutional layer module, a regular residual module, and a bottleneck residual module; The step of performing image recognition on the current driving image using a preset convolutional neural network model to obtain the image recognition result of the target object includes: The image features in the current driving image are downsampled sequentially through the single convolutional layer module, the conventional residual module, and the bottleneck residual module to obtain the target image features; The second stage involves predicting the image category based on the target image features to obtain the image recognition result of the target object.

2. The neural network-based object tracking method of claim 1, wherein, The second stage includes an average pooling layer, a fully connected layer, and a Softmax layer; The step of performing image category prediction based on the target image features in the second stage to obtain the image recognition result of the target object includes: The target image features are pooled using the average pooling layer to obtain the dimensionality-reduced image features. The fully connected layer is used to perform vector transformation on the dimensionality-reduced image features to obtain the image feature vector; The image recognition result of the target object is generated by the Softmax layer based on the image feature vector.

3. The neural network-based object tracking method of claim 1, wherein, Before the step of acquiring the current driving image captured by the dashcam, the method further includes: Obtain the initial convolutional neural network model to be trained, as well as the preset image training set; The initial convolutional neural network model is pre-trained based on the preset image training set to obtain a pre-trained convolutional neural network model. The pre-trained convolutional neural network model is used to initialize the convolutional neural network model for image recognition to obtain a preset convolutional neural network model.

4. The neural network-based object tracking method of claim 3, wherein, The step of initializing the convolutional neural network model for image recognition based on the pre-trained convolutional neural network model to obtain a preset convolutional neural network model includes: Obtain the model parameter weights corresponding to the pre-trained convolutional neural network model; The convolutional neural network model for image recognition is initialized according to the model parameter weights to obtain a preset convolutional neural network model.

5. The neural network-based object tracking method of claim 1, wherein, The step of determining whether the target object is a tracking object based on the image recognition result includes: Obtain the object type information and object location information of the target object from the image recognition results; Based on the object type information and the object location information, determine whether the target object is a tracking object.

6. The neural network-based object tracking method of claim 5, wherein, The step of determining whether the target object is a tracking object based on the object type information and the object location information includes: Determine whether the target object is located in the preset tracking area based on the object location information; When the target object is located in the preset tracking area, it is determined whether the target object is a tracking object based on the object type information.

7. The neural network-based object tracking method of claim 6, wherein, The step of determining whether the target object is located within the preset tracking area based on the object location information includes: Determine the boundary coordinates of the target object based on the object location information; The bounding box of the target object is determined based on the boundary coordinates; The target object is determined to be located within a preset tracking area based on the region where the bounding box is located in the current driving image.

8. The neural network-based object tracking method of claim 6, wherein, The step of determining whether the target object is a tracking object based on the object type information includes: Read the object type identifier from the object type information; The object type identifier is matched according to a preset tracking object list, and the target object is determined as a tracking object based on the matching result. The preset tracking object list stores the object type identifier corresponding to the tracking object.

9. The neural network-based object tracking method of any one of claims 1 to 8, wherein, Before the step of determining whether the target object is a tracking object based on the image recognition result, the method further includes: Obtain the previous frame of the current driving image; Detect whether the target object exists in the previous frame of the driving image; When the target object exists, the step of determining whether the target object is a tracking object based on the image recognition result is performed.

10. The object tracking method based on a neural network as described in claim 9, characterized in that, When the target object exists, the step of determining whether the target object is a tracking object based on the image recognition result includes: If the target object exists, obtain the distance between the target object and the target object; When the distance is less than the preset safe distance, the step of determining whether the target object is the tracking object based on the image recognition result is executed.

11. The neural network-based object tracking method of claim 10, wherein, The step of obtaining the distance to the target object when the target object exists includes: When the target object exists, a coordinate system is established using the camera corresponding to the dashcam; Based on the coordinate system, the distance between the object and the target object is obtained through perspective transformation.

12. The neural network-based object tracking method of claim 11, wherein, The coordinate system includes the world coordinate system and the camera coordinate system; The step of obtaining the distance between the target object and the coordinate system using perspective transformation technology includes: Based on the world coordinate system and the camera coordinate system, the distance between the target object and the target object is obtained through perspective transformation technology.

13. A neural network-based object tracking apparatus, characterized by comprising: The neural network-based image recognition device includes: The image acquisition module is used to acquire the current driving images captured by the dashcam; The image recognition module is used to perform image recognition on the current driving image through a preset convolutional neural network model to obtain the image recognition result of the target object. The preset convolutional neural network model includes a first stage for image feature extraction and a second stage for image category prediction. The first stage includes a convolutional layer with downsampling. The object determination module is used to determine whether the target object is a tracking object based on the image recognition result. An object tracking module is used to track the target object when the target object is a tracking object; The convolutional layer with downsampling includes a single convolutional layer module, a regular residual module, and a bottleneck residual module. The image recognition module is further used for: The image features in the current driving image are downsampled sequentially through the single convolutional layer module, the conventional residual module, and the bottleneck residual module to obtain the target image features; The second stage involves predicting the image category based on the target image features to obtain the image recognition result of the target object.

14. The neural network-based object tracking apparatus of claim 13, wherein, The convolutional layer with downsampling includes a single convolutional layer module, a regular residual module, and a bottleneck residual module; The image recognition module is further configured to downsample the image features in the current driving image sequentially through the single convolutional layer module, the conventional residual module, and the bottleneck residual module to obtain the target image features; The image recognition module is further configured to perform image category prediction based on the target image features in the second stage to obtain the image recognition result of the target object.

15. The neural network-based object tracking apparatus of claim 14, wherein, The second stage includes an average pooling layer, a fully connected layer, and a Softmax layer; The image recognition module is further configured to perform pooling operations on the target image features through the average pooling layer to obtain the dimensionality-reduced image features; The image recognition module is further configured to perform vector transformation on the dimensionality-reduced image features through the fully connected layer to obtain an image feature vector; The image recognition module is also used to generate an image recognition result of the target object based on the image feature vector through the Softmax layer.

16. The neural network-based object tracking apparatus of claim 13, wherein, The device also includes a model training module; The model training module is used to obtain the initial convolutional neural network model to be trained, as well as the preset image training set; The model training module is also used to pre-train the initial convolutional neural network model according to the preset image training set to obtain a pre-trained convolutional neural network model. The model training module is further configured to initialize the convolutional neural network model for image recognition based on the pre-trained convolutional neural network model, so as to obtain a preset convolutional neural network model.

17. The neural network-based object tracking apparatus of any one of claims 13 to 16, wherein, The object determination module is also used to obtain the previous frame of the driving image of the current driving image; The object determination module is also used to detect whether the target object exists in the previous frame of the driving image; The object determination module is further configured to, when the target object exists, perform the operation of determining whether the target object is a tracking object based on the image recognition result.

18. A neural network based object tracking device, comprising: The device includes: a memory, a processor, and a neural network-based object tracking program stored in the memory and executable on the processor, the neural network-based object tracking program being configured to implement the steps of the neural network-based object tracking method as described in any one of claims 1 to 12.

19. A storage medium, characterized by The storage medium stores a neural network-based object tracking program, which, when executed by a processor, implements the steps of the neural network-based object tracking method as described in any one of claims 1 to 12.

Images