Obstacle determination method, apparatus, device, and readable storage medium

Abstract

The application discloses an obstacle determination method and device, equipment and a readable storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a first frame image and a second frame image describing an environment in which a target vehicle is located; performing spatial feature extraction on the first frame image and the second frame image by using a self-encoder to obtain spatial features of the first frame image and the second frame image; performing time feature extraction on the spatial features of the first frame image and the second frame image by using a long short-term memory model to obtain time features of the first frame image and the second frame image; determining an image error between the first frame image and the second frame image based on the time features of the first frame image and the second frame image; and determining that an obstacle exists if the image error is greater than an error threshold. The different images of the environment in which the vehicle is located are compared globally from the images, and it is determined that an obstacle exists in the environment when the image content difference is large.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to an obstacle determination method and device, equipment and readable storage medium. BACKGROUND

[0002] With the development of computer technology, the types and quantities of vehicles are increasing, which enables more and more people to drive vehicles. In the process of driving, due to the existence of visual blind area or the phenomenon of inattention of the driver, traffic accidents occur more and more frequently, which brings great threat to people's life and property. Based on this, how to determine whether there is an obstacle in the environment where the vehicle is located to reduce the frequency of traffic accidents has become a problem to be solved. SUMMARY

[0003] The present application provides an obstacle determination method, device, equipment and readable storage medium, which can be used to solve the problems in the related art. The technical solution includes the following contents.

[0004] In one aspect, an obstacle determination method is provided, which includes:

[0005] obtaining a first frame image and a second frame image, the first frame image and the second frame image being different images for describing the environment where the target vehicle is located;

[0006] extracting spatial features of the first frame image by using an autoencoder to obtain spatial features of the first frame image, the spatial features of the first frame image being used to describe each first image block in the first frame image;

[0007] extracting time features of the first frame image by using a long short-term memory model to obtain time features of the first frame image, the time features of the first frame image being used to describe the association relationship between the each first image block;

[0008] extracting spatial features of the second frame image by using the autoencoder to obtain spatial features of the second frame image, the spatial features of the second frame image being used to describe each second image block in the second frame image;

[0009] extracting time features of the second frame image by using the long short-term memory model to obtain time features of the second frame image, the time features of the second frame image being used to describe the association relationship between the each second image block;

[0010] determining an image error between the first frame image and the second frame image based on the time features of the first frame image and the time features of the second frame image;

[0011] In a case where the image error is greater than an error threshold, it is determined that an environment in which the target vehicle is located has an obstacle.

[0012] In another aspect, an obstacle determination apparatus is provided, and the apparatus comprises:

[0013] An acquisition module is configured to acquire a first frame image and a second frame image, the first frame image and the second frame image being different images for describing an environment in which a target vehicle is located;

[0014] A spatial feature extraction module is configured to perform spatial feature extraction on the first frame image by using an auto-encoder to obtain spatial features of the first frame image, the spatial features of the first frame image being used to describe each first image block in the first frame image;

[0015] A temporal feature extraction module is configured to perform temporal feature extraction on the spatial features of the first frame image by using a long short-term memory model to obtain temporal features of the first frame image, the temporal features of the first frame image being used to describe an association relationship between the each first image block;

[0016] The spatial feature extraction module is further configured to perform spatial feature extraction on the second frame image by using the auto-encoder to obtain spatial features of the second frame image, the spatial features of the second frame image being used to describe each second image block in the second frame image;

[0017] The temporal feature extraction module is further configured to perform temporal feature extraction on the spatial features of the second frame image by using the long short-term memory model to obtain temporal features of the second frame image, the temporal features of the second frame image being used to describe an association relationship between the each second image block;

[0018] A determination module is configured to determine an image error between the first frame image and the second frame image based on the temporal features of the first frame image and the temporal features of the second frame image;

[0019] The determination module is further configured to determine that an environment in which the target vehicle is located has an obstacle in a case where the image error is greater than an error threshold.

[0020] In another aspect, an electronic device is provided, and the electronic device comprises a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to implement any of the above-described obstacle determination methods.

[0021] In another aspect, a computer readable storage medium is provided, wherein at least one computer program is stored in the computer readable storage medium, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above obstacle determination methods.

[0022] In another aspect, a computer program is provided, wherein the computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above obstacle determination methods.

[0023] In another aspect, a computer program product is provided, wherein at least one computer program is stored in the computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above obstacle determination methods.

[0024] The technical solutions provided in the present application at least bring the following beneficial effects:

[0025] The technical solutions provided in the present application at least bring the following beneficial effects: BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0027] Figure 1 is a schematic diagram of an implementation environment of an obstacle determination method provided by an embodiment of the present application;

[0028] Figure 2 is a flowchart of an obstacle determination method provided by an embodiment of the present application;

[0029] Figure 3 is a structural schematic diagram of a self-encoder provided by an embodiment of the present application.

[0030] Figure 4 is a flowchart of detecting an obstacle provided by an embodiment of the present application.

[0031] Figure 5 is a structural schematic diagram of an obstacle determination apparatus provided by an embodiment of the present application.

[0032] Figure 6 is a structural schematic diagram of a vehicle terminal provided by an embodiment of the present application.

[0033] Figure 7 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0035] Figure 1 is an implementation environment schematic diagram of an obstacle determination method provided by an embodiment of the present application, as shown in the figure, the implementation environment includes a vehicle terminal 11. Wherein, the obstacle determination method in the embodiments of the present application can be executed by the vehicle terminal 11. The vehicle terminal 11 can be a smart phone, a desktop computer, a tablet computer, a laptop computer, a smart vehicle device, a smart voice interaction device, etc. The number of vehicle terminals 11 is not limited, and can be one or more. Figure 1

[0036] Optionally, the implementation environment can also include a server, the obstacle determination method in the embodiments of the present application can be executed by the server, or jointly executed by the vehicle terminal 11 and the server. The server can be a server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server can be connected with the vehicle terminal 11 through a wired network or a wireless network. The server can have functions of data processing, data storage and data transceiving, which are not limited in the embodiments of the present application. The number of servers is not limited, and can be one or more.

[0037] With the development of computer technology, the types and quantities of vehicles are increasing, so that more and more people have the ability to drive vehicles. When there is an obstacle in the environment of the vehicle, the vehicle may collide with the obstacle, which brings great threat to people's life and property. Based on this, how to determine whether there is an obstacle in the environment of the vehicle becomes a problem to be solved.

[0038] ​The embodiment of the present application provides a kind of obstacle determination method, which can be applied to the implementation environment described above. Figure 2 As shown in the flow chart of the obstacle determination method provided by the embodiment of the present application, for the convenience of description, the vehicle terminal 11 or server executing the obstacle determination method in the embodiment of the present application is called electronic device, and the method can be executed by the electronic device.As shown in the flow chart of the obstacle determination method provided by the embodiment of the present application, for the convenience of description, the vehicle terminal 11 or server executing the obstacle determination method in the embodiment of the present application is called electronic device, and the method can be executed by the electronic device. Figure 2 As shown in the flow chart of the obstacle determination method provided by the embodiment of the present application, for the convenience of description, the vehicle terminal 11 or server executing the obstacle determination method in the embodiment of the present application is called electronic device, and the method can be executed by the electronic device.

[0039] Step 201, obtain the first frame image and the second frame image, and the first frame image and the second frame image are different images for describing the environment where the target vehicle is located.

[0040] In the embodiment of the present application, the target vehicle is any vehicle, and the target vehicle is configured with an image acquisition device. The environment where the target vehicle is located is collected by the image acquisition device in real time or periodically or randomly to obtain the environment image for describing the environment. The image acquisition device is not limited in the present application. For example, the image acquisition device can be a camera, a video camera, etc.

[0041] The environment image is multiple frames, and the first frame image and the second frame image can be selected from the multiple frames of environment images. It can be understood that there are many screening methods. For example, in implementation mode A1, any two frames of environment images are taken as the first frame image and the second frame image respectively. Alternatively, in implementation mode A2, for any two adjacent frames of environment images in the multiple frames of environment images, the image error between the two frames of environment images is determined based on the pixel values of each pixel point in the two frames of environment images, and if the image error is greater than a set threshold, the two frames of environment images are determined as the first frame image and the second frame image respectively. In this way, the first frame image and the second frame image with large content difference can be selected, which is beneficial to determine the suddenly appearing obstacle. Alternatively, the first frame image and the second frame image are selected from the multiple frames of environment images according to the implementation mode A3 as shown below.

[0042] Implementation mode A3, step 201 includes steps 2011 to 2013.

[0043] Step 2011, obtain the environment video collected by the camera of the target vehicle, and the environment video includes multiple frames of environment images.

[0044] In the embodiment of the present application, the image acquisition device includes a camera, and the environment where the target vehicle is located is collected in real time by the camera to obtain a video. The environment video with video duration T or image frame number T can be extracted from the video, and the environment video includes multiple frames of images, and each frame of image is an environment image.

[0045] Step 2012, normalize each frame of environment image to obtain each frame of normalized image.

[0046] In the embodiments of the present application, any normalization algorithm can be used to perform normalization processing on any frame of environment image to obtain a normalized environment image, which is also referred to as a normalized image. The normalization algorithm includes but is not limited to L1 normalization algorithm, L2 normalization algorithm, gray normalization algorithm, etc.

[0047] For the L1 normalization algorithm, the sum of pixel values of each pixel point in any frame of environment image can be calculated, and the pixel value of any pixel point is divided by the sum of pixel values of each pixel point to obtain the normalized pixel value of any pixel point. In this way, the normalized pixel value of each pixel point is determined, and thus the normalized image is obtained. That is, the pixel value of each pixel point in the normalized image is the normalized pixel value of each pixel point.

[0048] For the L2 normalization algorithm, the square root of the sum of pixel values of each pixel point in any frame of environment image can be calculated, and the pixel value of any pixel point is divided by the square root of the sum of pixel values of each pixel point to obtain the normalized pixel value of any pixel point. In this way, the normalized pixel value of each pixel point is determined, and thus the normalized image is obtained. That is, the pixel value of each pixel point in the normalized image is the normalized pixel value of each pixel point.

[0049] For the gray normalization algorithm, the normalization processing can be performed according to steps 20121 to 20123 mentioned below. That is, step 2012 includes steps 20121 to 20123.

[0050] In step 20121, the gray conversion is performed on each frame of environment image to obtain each frame of gray image.

[0051] In the embodiments of the present application, any frame of environment image is a color image, and the pixel value of any pixel point in the color image includes the value of red (Red, R) channel, the value of green (Green, G) channel and the value of blue (Blue, B) channel. The gray conversion can be performed on any frame of environment image based on the three channel values of each pixel point in the environment image to obtain a gray image.

[0052] Optionally, the maximum value method is used for the gray scale conversion. That is, for any pixel point in any frame of the environment image, the maximum channel value is determined from the three channel values of the pixel point, and the channel value is taken as the gray scale value of the pixel point in the gray scale image, so as to obtain the gray scale image, and the gray scale image includes the gray scale values of the pixel points. Alternatively, the average value method or the weighted average value method is used for the gray scale conversion. That is, for any pixel point in any frame of the environment image, the three channel values of the pixel point are averaged or weighted averaged, and the calculated channel value is taken as the gray scale value of the pixel point in the gray scale image, so as to obtain the gray scale image.

[0053] In step 20122, based on the gray scale images, an average image is determined, and the gray scale value of any pixel point on the average image is obtained by averaging the gray scale values of any pixel point in the gray scale images.

[0054] In the embodiments of the present application, any two frames of the gray scale images include the same number of pixel points and the same arrangement of the pixel points. Based on this, the gray scale images include the same pixel points, and the gray scale values of the same pixel point in different gray scale images are the same or different. For example, each frame of the gray scale image includes N×N pixel points, and the N×N pixel points can be regarded as N rows and N columns of pixel points. Each frame of the gray scale image includes the i-th (i is a positive integer less than or equal to N) row and the j-th (j is a positive integer less than or equal to N) column of pixel points.

[0055] For any pixel point, the gray scale values of the pixel point in the gray scale images are averaged, and the calculated gray scale value is taken as the gray scale value of the pixel point in the average image, so as to obtain the average image.

[0056] In step 20123, based on the average image, the gray scale images are normalized to obtain normalized images.

[0057] In the embodiments of the present application, any frame of the gray scale image can be normalized based on the average image to obtain a normalized gray scale image, which is also called a normalized image.

[0058] For example, step 20123 includes: determining the average gray scale value of the average image and the gray scale value variance of the average image based on the gray scale values of the pixel points in the average image; for any frame of the gray scale image, determining the average gray scale value of any frame of the gray scale image and the gray scale value variance of any frame of the environment image based on the gray scale values of the pixel points in any frame of the gray scale image; and based on the average gray scale value of the average image, the gray scale value variance of the average image, the average gray scale value of any frame of the gray scale image, and the gray scale value variance of any frame of the gray scale image, normalizing any frame of the gray scale image to obtain a normalized image.

[0059] In the embodiments of the present application, the gray scale values of each pixel point in the average image can be averaged to obtain the average gray scale value of the average image. Then, for any pixel point in the average image, the square of the difference between the gray scale value of the pixel point in the average image and the average gray scale value of the average image is calculated. For the sake of description, the square is referred to as the square corresponding to the pixel point in the average image. The sum of the squares corresponding to each pixel point in the average image is divided by the number of pixel points in the average image to obtain the gray scale value variance of the average image.

[0060] Similarly, for any frame of gray scale image, the gray scale values of each pixel point in the gray scale image can be averaged to obtain the average gray scale value of the gray scale image. Then, for any pixel point in the gray scale image, the square of the difference between the gray scale value of the pixel point in the gray scale image and the average gray scale value of the gray scale image is calculated. For the sake of description, the square is referred to as the square corresponding to the pixel point in the gray scale image. The sum of the squares corresponding to each pixel point in the gray scale image is divided by the number of pixel points in the gray scale image to obtain the gray scale value variance of the gray scale image.

[0061] Next, the normalized image is determined according to formula (1) as follows.

[0062]

[0063]

[0064] Wherein, N(i,j) represents the gray scale value of the pixel point in the i-th row and the j-th column of the normalized image. i and j are positive integers. M0 represents the average gray scale value of the average image. V0 represents the gray scale value variance of the average image. I(i,j) represents the gray scale value of the pixel point in the i-th row and the j-th column of any frame of gray scale image. M represents the average gray scale value of any frame of gray scale image. V represents the gray scale value variance of any frame of gray scale image.

[0065] In the manner of formula (1), the gray scale values of each pixel point in the normalized image corresponding to any frame of gray scale image can be determined, and the normalized image corresponding to the frame of gray scale image is determined. In this way, the normalized images corresponding to each frame of gray scale image can be determined, and multiple frames of normalized images are obtained.

[0066] Step 2013, screening the first frame of image and the second frame of image from the multiple frames of normalized images.

[0067] In the embodiments of the present application, any two frames of normalized images in the frames of normalized images can be respectively taken as the first frame image and the second frame image. Alternatively, for any two adjacent frames of normalized images in the frames of normalized images, the image error between the two frames of normalized images is determined based on the gray values of each pixel point in the two frames of normalized images, and if the image error is greater than a set threshold, the two frames of normalized images are respectively determined as the first frame image and the second frame image. In this way, the first frame image and the second frame image with large content difference can be screened out, which is beneficial to determining the suddenly appearing obstacle.

[0068] By performing the gray scale conversion and the normalization processing on the frames of environment images, the contrast of the images is enhanced, so that when the first frame image and the second frame image are screened from the frames of normalized images, the accuracy of the first frame image and the second frame image can be improved, and the number of images is reduced, and the determination efficiency of the obstacle is improved.

[0069] In step 202, spatial feature extraction is performed on the first frame image by using the autoencoder to obtain the spatial feature of the first frame image, and the spatial feature of the first frame image is used to describe each first image block in the first frame image.

[0070] In the embodiments of the present application, the first frame image includes a plurality of first image blocks. Any first image block includes a plurality of continuous pixel points in the first frame image, and any two adjacent first image blocks can include at least one same pixel point or can not include the same pixel point. The spatial feature of the first frame image can be obtained by performing spatial feature extraction on the first frame image by using the autoencoder, so as to describe each first image block by using the spatial feature.

[0071] In the embodiments of the present application, the structure, size, parameters and the like of the autoencoder are not limited. For example, the autoencoder can be a Tensorflow model, which mainly uses a tf.nn.conv2d function (a convolution function) to perform spatial feature extraction, and includes an input layer, a hidden layer and an output layer. Please refer to Figure 3 , Figure 3 is a structural schematic diagram of an autoencoder provided in the embodiments of the present application, which includes an input layer, a hidden layer and an output layer. The input layer is used to receive input data and transmit the input data to the hidden layer, the hidden layer is used to perform convolution processing on the input data, and the output layer is used to map the features output by the hidden layer into the spatial feature of the first frame image.

[0072] In a possible implementation manner, step 202 includes steps 2021 and 2022.

[0073] In step 2021, for any first image block in the first frame image, feature extraction is performed on the first image block by the auto-encoder to obtain a first feature of the first image block.

[0074] In the embodiment of the present application, the value of any pixel point in the first frame image is a gray value or a pixel value. The auto-encoder determines the updated value of the original pixel point based on the value of the original pixel point and the values of the surrounding pixel points. The original pixel point is at least one, and the first feature of the first image block includes the updated value of each original pixel point.

[0075] Optionally, for any pixel point in the first image block, the auto-encoder performs weighted summation calculation or weighted average calculation on the value of the pixel point and the values of the surrounding pixel points, and takes the calculation result or the sum of the calculation result and a bias term as the updated value of the pixel point. The updated value of at least one pixel point is used to determine the first feature of the first image block. In this way, the first feature of each first image block can be determined.

[0076] The auto-encoder can be regarded as a series of moving windows. The moving window corresponds to a correlation coefficient or a weight, and the moving window corresponds to a moving step. The moving window is moved on the first frame image, and the region of the first frame image covered by the moving window is a first image block. Based on the values of the pixel points in the first image block and the correlation coefficient or the weight corresponding to the moving window, the first feature of the first image block is determined. Then, the moving window is constantly moved on the first frame image according to the moving step, and the first feature of the first image block covered by the moving window is determined until the moving window moves out of the first frame image. From then on, the first feature of each first image block can be obtained.

[0077] In step 2022, the spatial feature of the first frame image is determined based on the first feature of each first image block.

[0078] That is, the spatial feature of the first frame image includes the first feature of each first image block. Optionally, each first image block included in the first frame image is denoted as x1, x2, x3, …, xn. n After feature extraction of each first image block by the auto-encoder, the spatial feature of the first frame image can be obtained, and the spatial feature of the first frame image includes the first feature h1, h2, h3, …, hn of each first image block. n

[0079] In step 203, the spatial feature of the first frame image is subjected to time feature extraction by a long short-term memory model to obtain a time feature of the first frame image, and the time feature of the first frame image is used to describe the association relationship between the first image blocks.

[0080] ​In the embodiments of the present application, the spatial features of the first frame of images include first features of each first image block. The first features of each first image block can be subjected to time feature extraction by a Long-Short Term Memory (LSTM) model to obtain time features of the first frame of images, so as to describe the correlation between each first image block by the time features, thereby enabling the time features to accurately describe the first frame of images from a global perspective.

[0081] The embodiments of the present application do not limit the structure, size, parameters, etc. of the Long-Short Term Memory model. Exemplarily, the Long-Short Term Memory model includes a plurality of convolutional layers. The first features h1, h2, h3, …, h n As the input of the Long-Short Term Memory model, the first features h1, h2, h3, …, h n are subjected to convolution processing to extract more deep-level feature information. Optionally, the convolution processing is performed by a convolution function, which is constructed based on a first function and a second function. The convolution function can be represented as f(x)*g(x)=∫f(m)·g(x-m)dm, f(x) represents the first function, g(x) represents the second function denoted as g function, * is a convolution symbol, · is a dot multiplication symbol, and ∫f(m)·g(x-m)dm represents the integration of f(m)·g(x-m) on m. As can be seen from the convolution function, convolution processing is the core of the calculation process of smoothing or sharpening, and can continuously extract more deep-level feature information.

[0082] In a possible implementation, step 203 includes steps 2031 to 2033.

[0083] In step 2031, for the first first image block, the first feature of the first first image block is subjected to feature extraction by the Long-Short Term Memory model to obtain the second feature of the first first image.

[0084] As mentioned above, the first features h1, h2, h3, …, h n of each first image block are taken as the input of the Long-Short Term Memory model. For the first feature h1 of the first first image block, h1 is subjected to convolution processing by the Long-Short Term Memory model, and more deep-level feature information is extracted by the convolution processing to obtain the second feature of the first first image. The convolution processing of h1 by the convolution function mentioned above will not be described herein again.

[0085] In step 2032, for any first image block other than the first first image block, the first feature of the any first image block and the second feature of the last first image block of the any first image block are subjected to feature extraction by the Long-Short Term Memory model to obtain the second feature of the any first image.

[0086] After obtaining the second feature of the first image, the second feature of the first image and the first feature h2 of the second first image block can be spliced to obtain the spliced feature corresponding to the second first image block. The spliced feature corresponding to the second first image block is processed by convolution through the long short-term memory model. Through the convolution processing, deeper feature information is extracted to obtain the second feature of the second first image. The convolution function mentioned above can be used to process the convolution of the spliced feature corresponding to the third first image block, and details are not repeated here.

[0087] Similarly, after obtaining the second feature of the second first image, the second feature of the second first image and the first feature h3 of the third first image block can be spliced to obtain the spliced feature corresponding to the third first image block. The spliced feature corresponding to the third first image block is processed by convolution through the long short-term memory model. Through the convolution processing, deeper feature information is extracted to obtain the second feature of the third first image. The convolution function mentioned above can be used to process the convolution of the spliced feature corresponding to the third first image block, and details are not repeated here.

[0088] In the above manner, the first feature of any first image block other than the first first image block is spliced with the second feature of the previous first image block of any first image block to obtain the spliced feature corresponding to any first image block. Then the spliced feature corresponding to any first image block is processed by convolution through the long short-term memory model. Through the convolution processing, deeper feature information is extracted to obtain the second feature of the first image.

[0089] Step 2033, determining the time feature of the first frame image based on the second feature of the last first image.

[0090] Through the manner of steps 2031 to 2032, the second features of the first to last first images can be obtained. The second feature of the last first image can be determined as the time feature of the first frame image, or the second features of the first to last first images can be determined as the time feature of the first frame image, or the fusion result obtained by fusing the second features of the first to last first images is determined as the time feature of the first frame image.

[0091] Step 204, extracting the spatial feature of the second frame image through the auto-encoder to obtain the spatial feature of the second frame image, and the spatial feature of the second frame image is used to describe each second image block in the second frame image.

[0092] In the embodiment of the present application, the second frame image includes a plurality of second image blocks. Any second image block includes a plurality of continuous pixel points in the second frame image, and any two adjacent second image blocks can include at least one same pixel point or can not include the same pixel point. The spatial feature of the second frame image can be extracted by the autoencoder to obtain the spatial feature of the second frame image, so as to describe each second image block by using the spatial feature. The implementation of step 204 can be found in the description of step 202, and the implementation principles are similar, and thus will not be described here.

[0093] In step 205, the spatial feature of the second frame image is extracted by using the long-short term memory model to obtain the time feature of the second frame image, and the time feature of the second frame image is used to describe the association relationship between each second image block.

[0094] In the embodiment of the present application, the spatial feature of the second frame image includes the first feature of each second image block. The first feature of each second image block can be extracted by using the long-short term memory (LSTM) model to obtain the time feature of the second frame image, so as to describe the association relationship between each second image block by using the time feature, thereby enabling the time feature to accurately describe the second frame image from a global perspective. The implementation of step 205 can be found in the description of step 203, and the implementation principles are similar, and thus will not be described here.

[0095] In step 206, the image error between the first frame image and the second frame image is determined based on the time feature of the first frame image and the time feature of the second frame image.

[0096] In the embodiment of the present application, the feature distance between the time feature of the first frame image and the time feature of the second frame image can be calculated according to the distance algorithm, and the feature distance is taken as the image error between the first frame image and the second frame image. The calculation manner of the distance algorithm is not limited in the embodiment of the present application, and the distance algorithm includes, but is not limited to, the cross-entropy algorithm, the relative entropy algorithm, and the like.

[0097] By calculating the feature distance between the time feature of the first frame image and the time feature of the second frame image, the image error between the first frame image and the second frame image is calculated at the feature level, so that the image error can reflect the content difference between the first frame image and the second frame image from the global image, and the accuracy of the image error is improved.

[0098] In a possible implementation manner, step 206 includes steps 2061 to 2063.

[0099] In step 2061, the time feature of the first frame image is decoded to obtain a first reconstructed image.

[0100] In the embodiment of the present application, the time feature of the first frame image can be decoded by a decoder. The decoder includes a deconvolution layer, and the time feature of the first frame image is deconvoluted by the deconvolution layer to reconstruct an image based on the time feature of the first frame image, thereby obtaining a first reconstructed image. The structure, size, parameters, etc. of the decoder are not limited in the embodiment of the present application. For example, the decoder is a Tensorflow model, which mainly uses a tf.nn.conv2d_transpose function (a kind of deconvolution function) to perform deconvolution processing.

[0101] The deconvolution processing is equivalent to automatic padding processing and convolution processing. Generally, the time feature of the first frame image is first subjected to automatic padding processing to obtain the time feature of the first frame image after padding, and then the time feature of the first frame image after padding is subjected to convolution processing to obtain the first reconstructed image. In the embodiment of the present application, the automatic padding processing is to supplement 0 in the time feature of the first frame image. Through the automatic padding processing and then the convolution processing, the size of the first reconstructed image can be consistent with the size of the first frame image.

[0102] In step 2062, the time feature of the second frame image is decoded to obtain a second reconstructed image.

[0103] In the embodiment of the present application, the time feature of the second frame image can be decoded by a decoder. The implementation principle of step 2062 is similar to that of step 2061, and the description of step 2061 can be referred to.

[0104] In step 2063, based on the first reconstructed image and the second reconstructed image, an image error between the first frame image and the second frame image is determined.

[0105] In the embodiment of the present application, the size of the first reconstructed image is consistent with the size of the first frame image, the size of the second reconstructed image is consistent with the size of the second frame image, and the size of the first frame image is consistent with the size of the second frame image. Therefore, the size of the first reconstructed image is consistent with the size of the second reconstructed image. That is, the first reconstructed image and the second reconstructed image include the same number of pixel points and have the same arrangement of pixel points, and the value of the same pixel point in the first reconstructed image and the second reconstructed image is the same or different. In other words, if the first reconstructed image and the second reconstructed image both include N×N pixel points, and the N×N pixel points are regarded as N rows and N columns of pixel points, then the first reconstructed image and the second reconstructed image both include the i-th (i is a positive integer less than or equal to N) row and j-th (j is a positive integer less than or equal to N) column of pixel points.

[0106] For any pixel point, subtract the value of the pixel point in the second reconstructed image from the value of the pixel point in the first reconstructed image to obtain a difference value corresponding to the pixel point. Calculate the sum, average, variance, standard deviation, or the like of the difference values corresponding to the respective pixel points, and take the calculation result as the image error between the first frame image and the second frame image.

[0107] By decoding the temporal features of the first frame image into the first reconstructed image and decoding the temporal features of the second frame image into the second reconstructed image, the image error is calculated at the pixel level based on the first reconstructed image and the second reconstructed image. The temporal features of the first frame image can globally describe the first frame image, so that the first reconstructed image determined based on the temporal features of the first frame image can also reflect the content of the first frame image. Similarly, the second reconstructed image can also reflect the content of the second frame image, so that the image error can intuitively reflect the content difference between the first frame image and the second frame image, and the accuracy of the image error is improved.

[0108] In step 207, if the image error is greater than the error threshold, it is determined that the environment in which the target vehicle is located has an obstacle.

[0109] The embodiments of the present application do not limit the determination manner of the error threshold. For example, the error threshold is a value set according to artificial experience, or the error threshold is a value obtained through experimental verification.

[0110] If the image error between the first frame image and the second frame image is greater than the error threshold, it indicates that the content difference between the first frame image and the second frame image is large, and new image content appears. Since the first frame image and the second frame image are both used to describe the environment in which the target vehicle is located, the appearance of new image content is equivalent to the appearance of an obstacle in the environment in which the target vehicle is located. That is to say, if the image error between the first frame image and the second frame image is greater than the error threshold, it is determined that the environment in which the target vehicle is located has an obstacle. Conversely, if the image error between the first frame image and the second frame image is not greater than the error threshold, it indicates that no new image content appears, so it can be determined that no new obstacle is added to the environment in which the target vehicle is located.

[0111] When it is determined that the environment in which the target vehicle is located has an obstacle, the distance between the target vehicle and the obstacle can be obtained. If the distance is greater than a threshold, a prompt information is sent to prompt the driving object to pay attention to the obstacle in the environment in which the target vehicle is located. The prompt information can be a sound or an image. If the distance is less than or equal to the threshold, the target vehicle is automatically braked to avoid collision between the target vehicle and the obstacle, and to affect the life and property safety of the driving object.

[0112] In summary, please refer to Figure 4 , Figure 4is a flowchart of detecting an obstacle provided by an embodiment of the present application. In the embodiment of the present application, the environment video can be preprocessed to obtain a first frame image and a second frame image. The environment video includes a plurality of environment images. Each environment image is first converted into grayscale and then normalized to obtain a normalized image corresponding to each environment image. Then, the first frame image and the second frame image are selected from the normalized images corresponding to each environment image. The implementation of this part can be seen from the description of step 201, and will not be repeated here.

[0113] Next, the spatial features of the first frame image are extracted by the autoencoder to obtain the spatial features of the first frame image, and the time features of the spatial features of the first frame image are extracted by the long short-term memory model to obtain the time features of the first frame image. Then, the time features of the first frame image are decoded to obtain the first reconstructed image. Similarly, the spatial features of the second frame image are extracted by the autoencoder to obtain the spatial features of the second frame image, and the time features of the spatial features of the second frame image are extracted by the long short-term memory model to obtain the time features of the second frame image. Then, the time features of the second frame image are decoded to obtain the second reconstructed image. This part can be seen from the description of steps 202 to 205, and will not be repeated here.

[0114] Then, the image error between the first reconstructed image and the second reconstructed image is calculated. Based on the image error, the obstacle detection result is determined. If the image error is greater than the error threshold, it is determined that the obstacle detection result is that there is an obstacle. This part can be seen from the description of steps 206 to 207, and will not be repeated here.

[0115] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in relevant regions. For example, the environment video, the first frame image and the second frame image involved in the present application are obtained under sufficient authorization.

[0116] The method realizes division of the images into image blocks from a spatial level by performing spatial feature extraction on the first frame image and the second frame image, and describes the image blocks by spatial features. Then, the spatial features of the first frame image and the spatial features of the second frame image are subjected to time feature extraction, thereby realizing association of the image blocks and describing the association relationship between the image blocks by time features, so that the time features can describe the image content from the global image and have strong representation ability. Subsequently, based on the time features of the first frame image and the time features of the second frame image, the image error between the first frame image and the second frame image is determined, and in the case that the image error is greater than an error threshold, it is determined that the environment where the target vehicle is located has an obstacle, thereby realizing comparison of different images of the environment where the target vehicle is located from the global image, and when the image content difference is large, it is determined that there is an obstacle in the environment, thereby improving the determination efficiency and accuracy of the obstacle.

[0117] Figure 5 Fig. 1 shows a structural schematic diagram of an obstacle determination apparatus provided by an embodiment of the present application, as shown in the figure, the apparatus comprises: Figure 5

[0118] The acquisition module 501 is configured to acquire a first frame image and a second frame image, the first frame image and the second frame image being different images used for describing an environment where a target vehicle is located;

[0119] The spatial feature extraction module 502 is configured to perform spatial feature extraction on the first frame image by using a self-encoder to obtain spatial features of the first frame image, the spatial features of the first frame image being used for describing first image blocks in the first frame image;

[0120] The time feature extraction module 503 is configured to perform time feature extraction on the spatial features of the first frame image by using a long short-term memory model to obtain time features of the first frame image, the time features of the first frame image being used for describing an association relationship between the first image blocks;

[0121] The spatial feature extraction module 502 is further configured to perform spatial feature extraction on the second frame image by using the self-encoder to obtain spatial features of the second frame image, the spatial features of the second frame image being used for describing second image blocks in the second frame image;

[0122] The time feature extraction module 503 is further configured to perform time feature extraction on the spatial features of the second frame image by using the long short-term memory model to obtain time features of the second frame image, the time features of the second frame image being used for describing an association relationship between the second image blocks;

[0123] The determination module 504 is configured to determine an image error between the first frame image and the second frame image based on the time features of the first frame image and the time features of the second frame image; ​

[0124] The determining module 504 is further configured to determine that the environment in which the target vehicle is located has an obstacle when the image error is greater than the error threshold.

[0125] In a possible implementation, the obtaining module 501 is configured to obtain an environment video collected by a camera of the target vehicle, the environment video including a plurality of environment images; perform normalization processing on each environment image to obtain a plurality of normalized images; and select a first image and a second image from the plurality of normalized images.

[0126] In a possible implementation, the obtaining module 501 is configured to perform grayscale conversion on each environment image to obtain a plurality of grayscale images; determine an average image based on the plurality of grayscale images, a grayscale value of any pixel point in the average image being obtained by averaging grayscale values of any pixel points in the plurality of grayscale images; and perform normalization processing on the plurality of grayscale images based on the average image to obtain a plurality of normalized images.

[0127] In a possible implementation, the obtaining module 501 is configured to determine an average grayscale value of the average image and a grayscale value variance of the average image based on grayscale values of each pixel point in the average image; for any grayscale image, determine an average grayscale value of the any grayscale image and a grayscale value variance of the any grayscale image based on grayscale values of each pixel point in the any grayscale image; and perform normalization processing on the any grayscale image based on the average grayscale value of the average image, the grayscale value variance of the average image, the average grayscale value of the any grayscale image, and the grayscale value variance of the any grayscale image to obtain a normalized image.

[0128] In a possible implementation, the first image includes a plurality of first image blocks; and the spatial feature extraction module 502 is configured to, for any first image block in the first image, perform feature extraction on the any first image block by using a self-encoder to obtain first features of the any first image block; and determine a spatial feature of the first image based on the first features of each first image block.

[0129] In a possible implementation, the spatial feature of the first image includes the first features of the plurality of first image blocks; and the temporal feature extraction module 503 is configured to, for a first first image block, perform feature extraction on the first features of the first first image block by using a long short-term memory model to obtain second features of the first first image; for any first image block other than the first first image block, perform feature extraction on the first features of the any first image block and second features of a previous first image block of the any first image block by using the long short-term memory model to obtain second features of the any first image; and determine a temporal feature of the first image based on the second features of a last first image.

[0130] In a possible implementation, the determining module 504 is configured to decode the time feature of the first frame image to obtain a first reconstructed image, decode the time feature of the second frame image to obtain a second reconstructed image, and determine the image error between the first frame image and the second frame image based on the first reconstructed image and the second reconstructed image.

[0131] The device described above extracts the spatial features of the first frame image and the second frame image, thereby dividing the image into image blocks from the spatial level and describing each image block by using the spatial features. Then, the spatial features of the first frame image and the second frame image are extracted to obtain the time features, thereby associating each image block and describing the association relationship between each image block by using the time features. The time features can describe the image content from the global image, and have strong representation ability. Subsequently, the image error between the first frame image and the second frame image is determined based on the time features of the first frame image and the second frame image. If the image error is greater than the error threshold, it is determined that the environment where the target vehicle is located has an obstacle, thereby comparing different images of the environment where the target vehicle is located from the global image. If the image content is significantly different, it is determined that there is an obstacle in the environment, and the determination efficiency and accuracy of the obstacle are improved.

[0132] It should be understood that the above Figure 5 The device provided in the present application is only exemplified by the above division of functional modules when realizing its functions. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be described here.

[0133] Figure 6 A structure block diagram of a vehicle terminal 600 provided in an example embodiment of the present application is shown. The vehicle terminal 600 includes a processor 601 and a memory 602.

[0134] The processor 601 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 601 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 601 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 601 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 601 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0135] The memory 602 can include one or more computer-readable storage media that can be non-transitory. The memory 602 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one computer program for being executed by the processor 601 to implement the obstacle determination method provided by the method embodiments in the present application.

[0136] In some embodiments, the vehicle terminal 600 can also optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602, and the peripheral device interface 603 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 603 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0137] The peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602 and the peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602 and the peripheral interface 603 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0138] The radio frequency circuit 604 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 604 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this regard.

[0139] The display screen 605 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 is further configured to capture touch signals on or above the surface of the display screen 605. The touch signals can be input to the processor 601 as control signals for processing. In this case, the display screen 605 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 605 can be one, disposed on the front panel of the vehicle terminal 600; in other embodiments, the display screen 605 can be at least two, respectively disposed on different surfaces of the vehicle terminal 600 or in a folding design; in other embodiments, the display screen 605 can be a flexible display screen, disposed on a curved surface or a folding surface of the vehicle terminal 600. Even, the display screen 605 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 605 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0140] The camera assembly 606 is configured to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function of the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 606 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0141] The audio circuit 607 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 601 for processing, or input to the radio frequency circuit 604 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the vehicle terminal 600. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 607 can also include a headphone jack.

[0142] The power supply 608 is used to supply power to each component in the vehicle terminal 600. The power supply 608 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 608 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0143] In some embodiments, the vehicle terminal 600 further includes one or more sensors 609. The one or more sensors 609 include, but are not limited to, an acceleration sensor 611, a gyroscope sensor 612, a pressure sensor 613, an optical sensor 614, and a proximity sensor 615.

[0144] The acceleration sensor 611 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the vehicle terminal 600. For example, the acceleration sensor 611 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for game or user motion data collection.

[0145] The gyroscope sensor 612 can detect the body direction and rotation angle of the vehicle terminal 600. The gyroscope sensor 612 can cooperate with the acceleration sensor 611 to collect 3D actions of the user on the vehicle terminal 600. The processor 601 can realize the following functions according to the data collected by the gyroscope sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0146] The pressure sensor 613 can be installed on the side bezel of the vehicle terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 613 is installed on the side bezel of the vehicle terminal 600, it can detect the user's grip signal on the vehicle terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is installed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0147] An optical sensor 614 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 614. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 614.

[0148] The proximity sensor 615, also known as a distance sensor, is typically installed on the front panel of the vehicle terminal 600. The proximity sensor 615 is used to detect the distance between the user and the front of the vehicle terminal 600. In one embodiment, when the proximity sensor 615 detects that the distance between the user and the front of the vehicle terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 615 detects that the distance between the user and the front of the vehicle terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0149] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the vehicle terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0150] Figure 7A server structure diagram provided by the embodiments of the present application is shown in FIG. 7. The server 700 can have great differences due to different configurations or performances, and can include one or more processors 701 and one or more memories 702, wherein the one or more memories 702 store at least one computer program, the at least one computer program is loaded and executed by the one or more processors 701 to implement the obstacle determination method provided by the above-mentioned various method embodiments. For example, the processor 701 is a CPU. Of course, the server 700 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, and the details are not described herein.

[0151] In exemplary embodiments, a computer readable storage medium is also provided, and the storage medium stores at least one computer program, the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned obstacle determination methods.

[0152] Optionally, the above-mentioned computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0153] In exemplary embodiments, a computer program is also provided, and the computer program is at least one, the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned obstacle determination methods.

[0154] In exemplary embodiments, a computer program product is also provided, and the computer program product stores at least one computer program, the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned obstacle determination methods.

[0155] It should be understood that "multiple" mentioned herein refers to two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0156] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0157] The above merely illustrates the embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining obstacles, characterized in that, The method includes: Acquire a first frame image and a second frame image, which are different images used to describe the environment in which the target vehicle is located; Spatial features of the first frame image are extracted by an autoencoder to obtain spatial features of the first frame image. The spatial features of the first frame image are used to describe each first image block in the first frame image. The spatial features of the first frame image include first features of multiple first image blocks. For the first image patch, the first feature of the first image patch is extracted using a long short-term memory model to obtain the second feature of the first image patch; For any first image block other than the first first image block, the first feature of the first image block and the second feature of the previous first image block of the first image block are concatenated to obtain the concatenated feature corresponding to the first image block. The concatenated feature corresponding to the first image block is then convolved by the long short-term memory model to obtain the second feature of the first image block. Based on the second feature of the last first image, the temporal feature of the first frame image is determined, and the temporal feature of the first frame image is used to describe the correlation between the various first image blocks. The second frame image is subjected to spatial feature extraction by the autoencoder to obtain the spatial features of the second frame image. The spatial features of the second frame image are used to describe each second image block in the second frame image. The spatial features of the second frame image are extracted using the Long Short-Term Memory model to obtain the temporal features of the second frame image. The temporal features of the second frame image are used to describe the correlation between the various second image blocks. Based on the temporal features of the first frame image and the temporal features of the second frame image, the image error between the first frame image and the second frame image is determined; If the image error is greater than the error threshold, it is determined that there is an obstacle in the environment where the target vehicle is located.

2. The method according to claim 1, characterized in that, The acquisition of the first frame image and the second frame image includes: Acquire environmental video captured by the camera of the target vehicle, the environmental video including multiple frames of environmental images; The environmental images of each frame are normalized to obtain normalized images of each frame; The first frame image and the second frame image are selected from the normalized images of each frame.

3. The method according to claim 2, characterized in that, The normalization process for each frame of the environmental image to obtain a normalized image for each frame includes: The environmental images of each frame are converted to grayscale to obtain grayscale images of each frame; Based on the grayscale images of each frame, an average image is determined. The grayscale value of any pixel in the average image is obtained by averaging the grayscale values ​​of any pixel in each frame of grayscale images. The average image is used to normalize the grayscale images of each frame to obtain normalized images of each frame.

4. The method according to claim 3, characterized in that, The step of normalizing the grayscale images of each frame based on the average image to obtain normalized images of each frame includes: Based on the gray values ​​of each pixel in the average image, the average gray value and the variance of the gray value of the average image are determined. For any frame of grayscale image, based on the grayscale value of each pixel in the grayscale image, determine the average grayscale value and the variance of the grayscale value of the grayscale image. Based on the average gray value of the average image, the gray value variance of the average image, the average gray value of any frame of grayscale image, and the gray value variance of any frame of grayscale image, the grayscale image of any frame is normalized to obtain a normalized image.

5. The method according to claim 1, characterized in that, The first frame image includes multiple first image blocks; the step of extracting spatial features from the first frame image using an autoencoder to obtain the spatial features of the first frame image includes: For any first image block in the first frame image, feature extraction is performed on the first image block by an autoencoder to obtain the first feature of the first image block; The spatial features of the first frame image are determined based on the first features of each first image block.

6. The method according to claim 1, characterized in that, The step of determining the image error between the first frame image and the second frame image based on the temporal features of the first frame image and the second frame image includes: The temporal features of the first frame image are decoded to obtain the first reconstructed image; The temporal features of the second frame image are decoded to obtain the second reconstructed image; Based on the first reconstructed image and the second reconstructed image, the image error between the first frame image and the second frame image is determined.

7. An obstacle detection device, characterized in that, The device includes: The acquisition module is used to acquire a first frame image and a second frame image, wherein the first frame image and the second frame image are different images used to describe the environment in which the target vehicle is located; The spatial feature extraction module is used to extract spatial features from the first frame image through an autoencoder to obtain the spatial features of the first frame image. The spatial features of the first frame image are used to describe each first image block in the first frame image. The spatial features of the first frame image include the first features of multiple first image blocks. The temporal feature extraction module is used to extract the first feature of the first image patch using a Long Short-Term Memory (LSTM) model to obtain the second feature of the first image patch; for any first image patch other than the first image patch, the first feature of the first image patch and the second feature of the previous first image patch are concatenated to obtain the concatenated feature corresponding to the first image patch; the concatenated feature corresponding to the first image patch is then convolved using the LSM model to obtain the second feature of the first image patch; based on the second feature of the last first image, the temporal feature of the first frame image is determined, and the temporal feature of the first frame image is used to describe the correlation between the various first image patches; The spatial feature extraction module is further configured to extract spatial features from the second frame image through the autoencoder to obtain the spatial features of the second frame image, and the spatial features of the second frame image are used to describe each second image block in the second frame image; The time feature extraction module is further used to extract time features from the spatial features of the second frame image through the long short-term memory model to obtain the time features of the second frame image. The time features of the second frame image are used to describe the correlation between the various second image blocks. The determining module is used to determine the image error between the first frame image and the second frame image based on the temporal features of the first frame image and the temporal features of the second frame image; The determining module is further configured to determine that there are obstacles in the environment where the target vehicle is located if the image error is greater than the error threshold.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the obstacle determination method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to implement the obstacle determination method as described in any one of claims 1 to 6.

Images