Method, device, terminal and storage medium for determining boundary of accessible space

Abstract

The present application discloses a method, device, terminal and storage medium for determining the boundary of a reachable space, and belongs to the field of image processing technology. Applied to a terminal, the method includes: extracting features from a first fisheye image to obtain a feature pyramid; obtaining the initial boundary of the reachable space in the first fisheye image based on the image features of the target layer in the feature pyramid and the regression model; obtaining a target heat map corresponding to the target layer and an offset map corresponding to the target layer based on the target layer and the heat map model, wherein the prediction range of the threshold function for generating the offset map in the heat map model is an integer; determining the actual boundary of the reachable space based on the initial boundary and the offset map. In the process of detecting the boundary of the reachable space of the first fisheye image, the present application combines the offset map obtained by combining the prediction range of the threshold function as an integer, so that the prediction range of the offset map is not limited by decimals, thereby improving the robustness of the neural network model for obtaining the boundary of the reachable space.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, terminal and storage medium for determining a boundary of a reachable space. Background Art

[0002] With the rapid development of science and technology, people have more and more demands on the functions of terminals, and the corresponding terminals can also realize more and more functions. For example, users can use terminals to perform space detection, audio playback, location positioning, etc.

[0003] Among them, in terms of space detection, cameras are usually used to collect environmental images of the surrounding environment, and the accessible space in the surrounding environment is detected based on the environmental images. At present, most of the neural network models used in terminals to detect accessible spaces mainly use global features and use the fully connected layer to output the boundary of the accessible space. In this process, if the number of specimens used by the neural network model is small, the predicted boundary of the accessible space will appear jagged, resulting in an uneven boundary of the accessible space and low robustness of the neural network model. Summary of the invention

[0004] The embodiments of the present application provide a method, device, terminal and storage medium for determining the boundary of a reachable space, which can improve the robustness of a neural network model for obtaining the boundary of a reachable space.

[0005] In one aspect, an embodiment of the present application provides a method for determining a boundary of a reachable space, which is applied to a terminal, and the method includes:

[0006] Performing feature extraction on the first fisheye image to obtain a feature pyramid, wherein the feature pyramid includes multiple layers of image features of the first fisheye image;

[0007] According to the image features of the target layer in the feature pyramid and the regression model, an initial boundary of the reachable space in the first fisheye image is obtained, wherein the target layer is a layer in the feature pyramid containing the largest number of features, and the regression model is a model trained based on at least two sample layers and obtaining an initial boundary with the same width as the sample layer;

[0008] According to the target layer and the heat map model, a target heat map corresponding to the target layer and an offset map corresponding to the target layer are obtained, wherein the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image;

[0009] An actual boundary of the reachable space is determined according to the initial boundary and the offset map.

[0010] Optionally, acquiring a target heat map corresponding to the target layer and an offset map corresponding to the target layer according to the target layer and the heat map model includes:

[0011] Extracting thermal features of the target layer through the thermal map model to obtain a thermal feature map, wherein image parameters of the thermal feature map are the same as those of the target layer, and the image parameters include width, length, and number of channels of pixel points;

[0012] Obtaining a thermal value from the thermal feature map through the thermal map model to obtain the target thermal map, wherein the target thermal map is a single-channel image, and the image size of the target thermal map is the same as the image size of the target layer;

[0013] The offset value of the thermal feature map is obtained through the thermal map model to obtain the offset map, which is a single-channel image. The image size of the offset map is the same as the image size of the target layer.

[0014] Optionally, after performing convolution processing on the thermal characteristic map through the third convolution layer to obtain the offset map, the method further includes:

[0015] In the offset map, determining a first pixel point corresponding to the initial boundary;

[0016] Expanding the pixel points included in the initial boundary in the offset image according to a preset expansion rule to obtain an expanded image area, wherein the preset expansion rule includes expanding N columns to both sides with the pixel point corresponding to the initial boundary as the center, where N is an integer greater than 0;

[0017] The determining, according to the initial boundary and the offset map, an actual boundary of the reachable space includes:

[0018] An actual boundary of the reachable space is determined according to the initial boundary and the expanded image region in the offset map.

[0019] Optionally, determining the actual boundary of the reachable space according to the initial boundary and the expanded image area in the offset map includes:

[0020] Determine each pixel point included in the expanded image area in the offset map;

[0021] The initial boundary and the pixel values ​​of each pixel point included in the expanded image area in the offset map are summed to determine the actual boundary of the reachable space.

[0022] Optionally, before determining the actual boundary of the reachable space according to the initial boundary and the offset map, the method further includes:

[0023] In the target heat map, determining a second pixel point corresponding to the initial boundary;

[0024] Obtaining the local maximum value of the pixel value corresponding to each pixel point within a preset image range of each second pixel point in the target heat map;

[0025] updating the initial boundary according to the positions of the pixels of each local maximum value in the first fisheye image;

[0026] The determining, according to the initial boundary and the offset map, an actual boundary of the reachable space includes:

[0027] The actual boundary of the reachable space is determined according to the updated initial boundary and the offset map.

[0028] Optionally, acquiring an initial boundary of a reachable space in the first fisheye image according to a target layer in the feature pyramid and a regression model includes:

[0029] Extracting the boundary of the target layer according to the regression model to obtain the boundary of the reachable space in the target layer;

[0030] The boundary of the reachable space in the target layer is used as the initial boundary of the reachable space in the first fisheye image, the initial boundary is a single-channel array, and the width of the initial boundary is the same as the width of the target layer.

[0031] Performing convolution processing on the target layer according to the fourth convolution layer to extract the regression feature map, wherein the image parameters of the regression feature map are the same as those of the target layer;

[0032] Convolutionally processing the regression feature map according to the fifth convolutional layer to obtain the initial boundary,

[0033] Optionally, the extracting features from the first fisheye image to obtain a feature pyramid includes:

[0034] Acquire a multi-layer feature image according to the first fisheye image and the target residual network;

[0035] The feature pyramid is generated according to the multi-layer feature image and the pyramid network.

[0036] In another aspect, an embodiment of the present application provides a device for determining a boundary of a reachable space, which is applied to a terminal, wherein the terminal includes a first timing device and a target interface, and the device includes:

[0037] An image acquisition module, used for performing feature extraction on the first fisheye image to obtain a feature pyramid, wherein the feature pyramid includes multiple layers of image features of the first fisheye image;

[0038] A boundary acquisition module, used to acquire an initial boundary of a reachable space in the first fisheye image according to image features of a target layer in the feature pyramid and a regression model, wherein the target layer is a layer in the feature pyramid containing the largest number of features, and the regression model is a model trained based on at least two sample layers and acquiring an initial boundary with the same width as the sample layer;

[0039] An offset map acquisition module is used to acquire a target heat map corresponding to the target layer and an offset map corresponding to the target layer according to the target layer and the heat map model, wherein the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image;

[0040] A boundary determination module is used to determine the actual boundary of the reachable space according to the initial boundary and the offset map.

[0041] On the other hand, an embodiment of the present application provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements a method for determining the boundary of a reachable space as described in one aspect above and any optional implementation method thereof.

[0042] In another aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the boundary of a reachable space as described in the above-mentioned other aspect and its optional manner.

[0043] The technical solution provided by the embodiments of the present application may at least include the following beneficial effects:

[0044] The terminal of the present application obtains a feature pyramid by extracting features from the first fisheye image; obtains the initial boundary of the reachable space in the first fisheye image based on the image features of the target layer in the feature pyramid and the regression model; obtains the target heat map corresponding to the target layer and the offset map corresponding to the target layer based on the target layer and the heat map model, the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image; determines the actual boundary of the reachable space based on the initial boundary and the offset map. In the process of detecting the boundary of the reachable space of the first fisheye image of the present application, the offset map obtained by combining the prediction range of the threshold function as an integer makes the prediction range of the offset map not limited by decimals, increases the output range of the offset map, makes the output boundary of the reachable space smoother, and improves the robustness of the neural network model for obtaining the boundary of the reachable space. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 is a schematic diagram of a fisheye image collected in a terminal according to an exemplary embodiment of the present application;

[0047] Figure 2 is a method flow chart of a method for determining a boundary of a reachable space provided by an exemplary embodiment of the present application;

[0048] Figure 3 is a method flow chart of a method for determining a boundary of a reachable space provided by an exemplary embodiment of the present application;

[0049] Figure 4 is an image schematic diagram of another target heat map involved in an exemplary embodiment of the present application;

[0050] Figure 5 is a schematic diagram of an initial boundary shown in an offset map according to an exemplary embodiment of the present application;

[0051] Figure 6 is a structural block diagram of a device for determining a boundary of a reachable space provided by an exemplary embodiment of the present application;

[0052] Figure 7 It is a structural diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0055] It should be noted that the terms "first", "second", "third" and "fourth" in the specification and claims of the present application are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present application and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] The solution provided by the present application can be used in scenarios where accessible spaces are detected by using a terminal in daily life. To facilitate understanding, some special terms and application architectures involved in the embodiments of the present application are briefly introduced below.

[0057] Neural Networks (NN) is a complex network system formed by a large number of simple processing units (called neurons) that are widely interconnected. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system. Neural networks have large-scale parallel, distributed storage and processing, self-organization, self-adaptation and self-learning capabilities, and are particularly suitable for dealing with imprecise and fuzzy information processing problems that require considering many factors and conditions at the same time. The development of neural networks is related to neuroscience, mathematical science, cognitive science, computer science, artificial intelligence, information science, cybernetics, robotics, microelectronics, psychology, optical computing, molecular biology, etc., and is an emerging frontier interdisciplinary subject.

[0058] Global features refer to the overall properties of an image. Common global features include color features, texture features, and shape features, such as intensity histograms. Since they are low-level visual features at the pixel level, global features have good invariance, simple calculations, and intuitive representations. However, their high feature dimensions and large amount of calculations are their fatal weaknesses. In addition, global feature descriptions are not suitable for image aliasing and occlusion. Local features are features extracted from local areas of an image, including edges, corners, lines, curves, and areas with special attributes. Common local features include two types of descriptions: corner point type and area type.

[0059] Rectified Linear Unit (ReLU), also known as the rectified linear unit, is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0060] In daily life, various terminals have been applied in various fields. For example, with the development of the vehicle field, vehicles are gradually moving towards autonomous driving. In the field of autonomous driving, the use of vehicle terminals is indispensable. Vehicles will encounter various obstacles, such as other vehicles, pedestrians, telephone poles, high-rise buildings and other obstacles. It is particularly important for cars to perceive the surrounding environment through vehicle terminals. In terms of environmental perception, data can be collected and calculated through hardware devices such as camera equipment and ultrasonic radar to achieve the effect of obstacle detection. For example, common camera equipment can include binocular cameras, monocular cameras, fisheye cameras and other devices.

[0061] Please refer to Figure 1 , which shows a schematic diagram of a fisheye image collected in a terminal according to an exemplary embodiment of the present application. Figure 1 As shown, the fisheye image 100 includes a first obstacle 101 and a second obstacle 102 .

[0062] Among them, the terminal in the present application is a terminal device that can be installed with the above-mentioned camera device, for example, a terminal that can be installed with a fisheye camera. The terminal can be a smart phone, a tablet computer, an e-book reader, smart glasses, a smart watch, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) player, a notebook computer, a laptop computer, a portable computer and a desktop computer, etc.

[0063] Optionally, the terminal can be an IoT device with an audio playback function, such as a sensor device, a computer of an IoT terminal, for example, a fixed, portable, pocket-sized, handheld, computer-built-in or vehicle-mounted device. For example, a station (STA), a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, an access point, a remote terminal, an access terminal, a user terminal, a user agent, a user device. Alternatively, the terminal can also be a device of an unmanned aerial vehicle. Alternatively, the terminal can also be a vehicle-mounted device, for example, a driving computer with an audio playback function, or a wireless communication device external to the driving computer.

[0064] In the above Figure 1 In the process, the terminal can trigger the function of detecting the reachable space of the fisheye image 100, wherein the reachable space generally refers to the space that can be reached in the image. For example, the vehicle-mounted terminal obtains the spatial range that it can move through the fisheye image, etc. Usually, the terminal can detect the reachable space in the fisheye image 100 through a pre-trained neural network model, obtain the global features corresponding to the fisheye image 100 (including the positions of the first obstacle 101 and the second obstacle 102), and directly output the ordinate of the reachable space based on the global features and using the fully connected layer. Optionally, the order in which the terminal outputs the ordinates can be arranged from small to large according to the abscissas in the fisheye image, and optionally, the number and value of the abscissas can be pre-fixed by the neural network model, for example, the value of the abscissa is (0, 1, ..., 319).

[0065] After obtaining the ordinate of the reachable space output by the fully connected layer, the terminal can also correct the ordinate of the reachable space in combination with the heat map to obtain a more accurate ordinate. At present, in the process of correction through the heat map, the prediction range of the threshold function used is between 0 and 1. When the number of annotations of the fisheye image collected by the neural network in the terminal is small, the boundary of the predicted reachable space is prone to jaggedness, making the accuracy of the ordinate of the corrected reachable space low, and there is a problem of low robustness of the neural network.

[0066] In order to improve the robustness of the neural network model for obtaining the boundary of the reachable space, the present application proposes a solution. In the process of obtaining the heat map by the heat map model, no restriction is imposed on the offset map of the redundant output, the prediction range of the threshold function for generating the offset map is set to an integer, the true value in the offset map is increased, and the robustness of the neural network after convergence is improved.

[0067] Please refer to Figure 2 , which shows a method flow chart of a method for determining the boundary of a reachable space provided by an exemplary embodiment of the present application. The method for determining the boundary of a reachable space can be applied to the above Figure 1 In the terminal of the scenario architecture shown in FIG. 1 , the method can be executed by the terminal. Figure 2 As shown, the method for determining the boundary of the reachable space may include the following steps.

[0068] Step 201 : extract features from the first fisheye image to obtain a feature pyramid, where the feature pyramid includes multiple layers of image features of the first fisheye image.

[0069] Optionally, the first fisheye image may be a fisheye image captured by a fisheye camera of the terminal, or the first fisheye image may be a normal image captured by a normal camera of the terminal and distorted to obtain a corresponding fisheye image. The present application does not limit the method for obtaining the first fisheye image.

[0070] Optionally, the terminal extracts features from the first fisheye image acquired by the terminal through a pre-designed neural network to generate a feature pyramid containing multiple layers of features, and each layer in the feature pyramid corresponds to features of different degrees of resolution. For example, the image parameters of the first fisheye image are (h1, w1, 3), where h1 represents the height of the first fisheye image, w1 represents the width of the first fisheye image, and 3 represents that each pixel has 3 channels. After extracting features from the first fisheye image, the features of each layer in the feature pyramid can be as follows: (h1 / 4, w1 / 4, 64), (h1 / 8, w1 / 8, 128), (h1 / 16, w1 / 16, 256). The image parameters of each layer of features are different from those of the first fisheye image.

[0071] Step 202: Acquire the initial boundary of the reachable space in the first fisheye image according to the image features of the target layer in the feature pyramid and the regression model.

[0072] The target layer is the layer with the largest number of features in the feature pyramid, and the regression model is a model that is trained based on at least two sample layers and obtains an initial boundary with the same width as the sample layer.

[0073] Optionally, the terminal inputs the image features of the target layer in the feature pyramid into the regression model, and uses the regression model to obtain the initial boundary of the reachable space in the first fisheye image. For example, the features of each layer obtained in the above feature pyramid are as follows: (h1 / 4, w1 / 4, 64), (h1 / 8, w1 / 8, 128), (h1 / 16, w1 / 16, 256). Then, the target layer is the layer corresponding to (h1 / 16, w1 / 16, 256).

[0074] Optionally, the regression model can be pre-trained by the developer and set in the terminal. For example, the developer uses at least two sample layers as sample data to train the regression model. After the first sample layer is input into the regression model, the regression model is used to obtain an initial boundary of the same width as the first sample layer. After the second sample layer is input into the regression model, the regression model is used to obtain an initial boundary of the same width as the second sample layer, and model training methods such as model convergence are performed to finally obtain a regression model that can obtain an initial boundary of the same width as the target layer when the input is the target layer. Among them, at least two sample layers can be obtained by developers extracting features from different fisheye images to obtain feature pyramids, and annotating the feature pyramids to obtain each sample layer.

[0075] Step 203, according to the target layer and the heat map model, obtain the target heat map corresponding to the target layer and the offset map corresponding to the target layer, the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image.

[0076] Among them, the terminal inputs the above-mentioned target layer into the heat map model, uses the heat map model to obtain the target heat map corresponding to the target layer, and outputs the offset map corresponding to the target layer. The pixel value of each pixel in the target heat map represents the heat of the point, and the pixel value of the pixel in the offset map represents the offset value of the corresponding position of the point in the first fisheye image. Optionally, the prediction range of the threshold function used in the process of generating the offset map in the heat map model is an integer greater than 1, such as 2, 3, 4, 5... and so on.

[0077] Step 204: Determine the actual boundary of the reachable space based on the initial boundary and the offset map.

[0078] Optionally, the terminal offsets the initial boundary according to the initial boundary obtained by the regression model and the offset map obtained by the heat map model, and obtains the offset boundary in the first fisheye image, thereby determining the actual boundary of the reachable space.

[0079] In summary, the terminal of the present application obtains a feature pyramid by performing feature extraction on the first fisheye image; obtains the initial boundary of the reachable space in the first fisheye image according to the image features of the target layer in the feature pyramid and the regression model; obtains the target heat map corresponding to the target layer and the offset map corresponding to the target layer according to the target layer and the heat map model, the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image; determines the actual boundary of the reachable space according to the initial boundary and the offset map. In the process of performing boundary detection of the reachable space on the first fisheye image of the present application, the offset map obtained by combining the prediction range of the threshold function as an integer makes the prediction range of the offset map not limited by decimals, increases the output range of the offset map, makes the output boundary of the reachable space smoother, and improves the robustness of the neural network model for obtaining the boundary of the reachable space.

[0080] In one possible implementation, the above-mentioned thermal map model includes extracting thermal features of the target layer to obtain a thermal feature map, and extracting thermal values ​​and offset values ​​based on the thermal feature map to obtain the corresponding target thermal map and offset map, and setting the prediction range of the threshold function used in the process of extracting the offset value to an integer, so as to achieve the effect of improving the robustness of the neural network model for obtaining the boundary of the reachable space.

[0081] Please refer to Figure 3 , which shows a method flow chart of a method for determining the boundary of a reachable space provided by an exemplary embodiment of the present application. The method for determining the boundary of a reachable space can be applied to the above Figure 1 In the terminal of the scenario architecture shown in FIG. 1 , the method can be executed by the terminal. Figure 3 As shown, the method for determining the boundary of the reachable space may include the following steps.

[0082] Step 301, obtaining a multi-layer feature image according to the first fisheye image and the target residual network.

[0083] Optionally, the acquisition of the first fisheye image can refer to the description in the above step 201, which will not be repeated here. In the present application, the terminal uses the target residual network to obtain a multi-layer feature image based on the acquired first fisheye image. The target residual network is a multi-layer feature pre-designed by the developer for extracting the first fisheye image. For example, the target residual network can be a Resnet34 residual network. Taking the image parameters of the first fisheye graphic as (h1, w1, 3) as an example, after the terminal processes the first fisheye image through the Resnet34 residual network, the multi-layer features obtained are as follows: (h1 / 4, w1 / 4, 64), (h1 / 8, w1 / 8, 128), (h1 / 16, w1 / 16, 256), (h1 / 32, w1 / 32, 512).

[0084] Step 302: Generate a feature pyramid based on the multi-layer feature image and the pyramid network.

[0085] The feature pyramid includes multiple layers of image features of the first fisheye image.

[0086] Optionally, the terminal generates a feature pyramid based on the multi-layer feature image obtained after processing the above-mentioned target residual network and the pyramid network. The pyramid network is the same as the target residual network, and is also pre-designed by the developer and set in the terminal. For example, the pyramid network can be a feature pyramid network (FPN), wherein FPN is a top-down feature fusion method, and FPN is a multi-scale target detection algorithm, that is, there is more than one feature prediction layer. Although some algorithms also use multi-scale feature fusion for target detection, they often only use features of one scale obtained after fusion. Although this approach can fuse the semantic information of the top-level features and the detail information of the bottom-level features, it will cause some deviations in the process of feature deconvolution, etc., and only using the features obtained after fusion for prediction will have an adverse effect on the detection accuracy. Starting from the above problem, the FPN method can make predictions on fused features of multiple different scales to maximize the detection accuracy.

[0087] Step 303, based on the image features of the target layer in the feature pyramid and the regression model, the initial boundary of the reachable space in the first fisheye image is obtained, the target layer is the layer with the largest number of features in the feature pyramid, and the regression model is a model that is trained based on at least two sample layers and obtains an initial boundary with the same width as the sample layer.

[0088] Optionally, after obtaining the feature pyramid, the terminal extracts a layer containing the largest number of features in the feature pyramid (i.e., the target layer), inputs the target layer into the regression model, and uses the regression model to obtain the initial boundary of the reachable space in the first fisheye image.

[0089] In one possible implementation, the terminal extracts the boundary of the target layer according to the regression model to obtain the boundary of the reachable space in the target layer; the boundary of the reachable space in the target layer is used as the initial boundary of the reachable space in the first fisheye image, the initial boundary is a single-channel array, and the width of the initial boundary is the same as the width of the target layer. For example, the terminal uses the first convolution layer and the second convolution layer in the regression model to process the target layer in sequence to extract the initial boundary of the reachable space in the target layer.

[0090] Optionally, taking the image parameters of the first fisheye image as (h1, w1, 3), after obtaining the corresponding feature pyramid, the image parameters of the target layer are (h1 / 4, w1 / 4, 128) as an example, the first convolutional layer can be: conv(128*64*3*3), where each number in conv represents: output channel, input channel, kernel width, kernel height. The terminal convolves the target layer with the first convolutional layer, and performs batch normalization (BN) and the first ReLU function on the convolution result to obtain a regression feature map, where the image parameters of the regression feature map are the same as those of the target layer. The image parameters of the regression feature map are also (h1 / 4, w1 / 4, 128).

[0091] Optionally, after obtaining the regression feature map in the regression model, the regression feature map is processed by the second convolution layer to extract the initial boundary of the reachable space in the target layer. Among them, the second convolution layer can be: conv(320*128*w1 / 4*h1 / 4), the regression model convolves the regression feature map obtained above with the second convolution layer, and performs BN and the second ReLU function on the convolution result in turn to obtain the initial boundary of the reachable space in the target layer, and the obtained initial boundary of the reachable space in the target layer is used as the initial boundary of the reachable space in the first fisheye image. Optionally, the initial boundary can be represented by y_init, then, based on the above example, y_init obtained in this step is a single-channel array, and the width of the initial boundary is the same as the width of the target layer, both of which are w1 / 4.

[0092] Step 304, according to the target layer and the heat map model, obtain the target heat map corresponding to the target layer and the offset map corresponding to the target layer, the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image.

[0093] In one possible implementation, the terminal extracts thermal features of the target layer through a thermal map model to obtain a thermal feature map. The image parameters of the thermal feature map are the same as those of the target layer, and the image parameters include width, length, and number of channels of pixel points. For example, the thermal map model also includes a third convolutional layer. After the terminal inputs the target layer into the thermal map model, the thermal map model uses the third convolutional layer to extract thermal features of the target layer and obtain thermal features.

[0094] Optionally, still taking the image parameters of the first fisheye image as (h1, w1, 3), after obtaining the corresponding feature pyramid, the image parameters of the target layer are (h1 / 4, w1 / 4, 128) as an example, the third convolutional layer can be: conv(128*64*3*3), wherein the terminal convolves the target layer with the third convolutional layer, and performs batch normalization (BatchNormalization, BN) and the third ReLU function on the convolution results in turn to obtain a thermal feature map, wherein the image parameters of the thermal feature map are the same as the image parameters of the target layer. That is, the image parameters of the thermal feature map are also (h1 / 4, w1 / 4, 128).

[0095] In one possible implementation, the terminal obtains the thermal value of the thermal feature map through the thermal map model to obtain a target thermal map, wherein the target thermal map is a single-channel image, and the image size of the target thermal map is the same as the image size of the target layer. Optionally, the thermal map model also includes a fourth convolutional layer. After the terminal obtains the thermal feature map through the thermal map model, the thermal feature map is convolved with the fourth convolutional layer to extract the thermal value, thereby obtaining the target thermal map.

[0096] Optionally, taking the above-mentioned thermal feature map image parameters of (h1 / 4, w1 / 4, 128) as an example, in this process, the fourth convolution layer can be: conv(1*128*3*3). The terminal convolves the thermal feature map with the fourth convolution layer, and performs batch normalization (Batch Normalization, BN) and the fourth ReLU function on the convolution results to obtain the target heat map, wherein the target heat map is a single-channel image, and the image size of the target heat map is the same as the image size of the target layer. For example, the obtained target heat map can be represented by heatmap(h1 / 4, w1 / 4).

[0097] In one possible implementation, the terminal obtains an offset value for the thermal feature map through a thermal map model to obtain an offset map, wherein the offset map is a single-channel image, and the image size of the offset map is the same as the image size of the target layer. Optionally, the thermal map model also includes a fifth convolutional layer. After the terminal obtains the thermal feature map through the thermal map model, the thermal feature map is convolved with the fifth convolutional layer to extract the offset value, thereby obtaining the offset map.

[0098] Optionally, taking the image parameters of the thermal feature map obtained above as (h1 / 4, w1 / 4, 128) as an example, in this process, the fifth convolutional layer can be: conv(1*128*3*3). The terminal convolves the thermal feature map with the fifth convolutional layer, and performs batch normalization (BN) and the fifth ReLU function on the convolution results to obtain an offset map, wherein the offset map is also a single-channel image, and the image size of the offset map is the same as the image size of the target layer. For example, the obtained offset map can be represented by offsetmap(h1 / 4, w1 / 4).

[0099] In one possible implementation, the terminal can also determine the second pixel points corresponding to the initial boundary in the target heat map; obtain the local maximum values ​​of the pixel values ​​corresponding to each pixel point within the preset image range of each second pixel point in the target heat map; and update the initial boundary according to the positions of the pixel points of each local maximum value in the first fisheye image. For example, after the terminal obtains the target heat map, the second pixel points corresponding to the initial boundary are determined in the target heat map, and the local maximum values ​​of the pixel values ​​corresponding to each pixel point within the preset image range of each second pixel point are obtained, and the second pixel points corresponding to the initial boundary in the first fisheye image are replaced with the pixel points whose pixel values ​​are local maximum values ​​within the preset image range.

[0100] The preset image range can be set by the developer in advance in the terminal. For example, please refer to Figure 4 , which shows an image schematic diagram of another target thermal map involved in an exemplary embodiment of the present application. Figure 4As shown, the target heat map 400 includes an initial boundary 401, a pixel point 1 402, a pixel point 2 403, and a preset image range 404. The terminal obtains the initial boundary in the first fisheye image through the above scheme, and determines the initial boundary 401 in the target heat map. The pixels included in the initial boundary 401 are the second pixels. Taking a pixel point (pixel point 1 402) among the second pixels as an example, the terminal can determine the pixels within the preset image range 404 according to pixel point 1 402, and calculate the local maximum for each pixel within the preset image range 404. Taking the pixel point corresponding to the local maximum as pixel point 2 403 as an example, the terminal replaces pixel point 1 402 with pixel point 2 403, thereby achieving an update effect, and updates the position of pixel point 2 403 corresponding to the first fisheye image to the initial boundary, thereby achieving a preliminary correction of the initial boundary.

[0101] Optionally, the above initial boundary is represented by y_init. The terminal finds the local maximum value of each pixel in y_init within a preset image range in the target heat map and updates y_init to obtain a more accurate initial boundary represented by y_hm.

[0102] Step 305: Determine the first pixel point corresponding to the initial boundary in the offset map.

[0103] Optionally, after acquiring the offset map through the above-mentioned heat map model, the terminal may determine the first pixel point corresponding to the initial boundary in the offset map.

[0104] Step 306: Expand the pixels included in the initial boundary in the offset image according to a preset expansion rule to obtain an expanded image area.

[0105] The preset expansion rule includes expanding N columns toward both sides with the pixel point corresponding to the initial boundary as the center, where N is an integer greater than 0.

[0106] Optionally, taking N=1 as an example, the row where each pixel point corresponding to the initial boundary determined in the offset map is located is expanded by one row upward, and the row where each pixel point corresponding to the initial boundary determined in the offset map is expanded by one row downward, thereby obtaining three rows of pixel points. In the offset map, the pixel points of these three rows are the pixel points corresponding to the initial boundary. For example, please refer to Figure 5 , which shows a schematic diagram of an initial boundary shown in an offset map according to an exemplary embodiment of the present application. Figure 5 As shown, the offset map 500 includes an initial boundary 501 and a first segment 502, where the first segment 502 is a portion of the initial boundary 501. The pixel value of the first segment 502 may be as follows:

[0107] [[-9.99e+02,-9.99e+02,-9.99e+02,-9.99e+02,-9.99e+02],

[0108] [1.97e+00, 1.97e+00, 1.97e+00, 1.97e+00, 1.97e+00],

[0109] [9.70e+00, 9.70e+00, 9.70e+00, 9.70e+00, 9.70e+00],

[0110] [-3.00e+00, -3.00e+00, -3.00e+00, -3.00e+00, -3.00e+00],

[0111] [-9.99e+00, -9.99e+00, -9.99e+00, -9.99e+00, -9.99e+00]]

[0112] A part of the initial boundary 501 corresponds to [9.70e+00, 9.70e+00, 9.70e+00, 9.70e+00, 9.70e+00] in the first segment 502. The terminal expands the pixel points of this row according to the preset expansion rule to obtain the pixel points of the previous row [1.97e+00, 1.97e+00, 1.97e+00, 1.97e+00, 1.97e+00] and the next row [-3.00e+00, -3.00e+00, -3.00e+00, -3.00e+00, -3.00e+00] and uses each pixel point as the pixel point included in the initial boundary 501 in the offset map.

[0113] Step 307: Determine the actual boundary of the reachable space according to the initial boundary and the expanded image area in the offset map.

[0114] In one possible implementation, the terminal determines each pixel point included in the expanded image area in the offset map; the initial boundary is summed with the pixel values ​​of each pixel point included in the expanded image area in the offset map to determine the actual boundary of the reachable space. That is, the obtained initial boundary y_init is summed with the pixel values ​​of each pixel point included in the expanded image area in the offset map to determine the actual boundary of the reachable space after the offset. In the above implementation, when the initial boundary y_init is updated to obtain a more accurate initial boundary represented by y_hm, the terminal can also use the more accurate y_hm and the expanded image area in the offset map to determine the actual boundary of the reachable space.

[0115] In summary, the terminal of the present application obtains a feature pyramid by performing feature extraction on the first fisheye image; obtains the initial boundary of the reachable space in the first fisheye image according to the image features of the target layer in the feature pyramid and the regression model; obtains the target heat map corresponding to the target layer and the offset map corresponding to the target layer according to the target layer and the heat map model, the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image; determines the actual boundary of the reachable space according to the initial boundary and the offset map. In the process of performing boundary detection of the reachable space on the first fisheye image of the present application, the offset map obtained by combining the prediction range of the threshold function as an integer makes the prediction range of the offset map not limited by decimals, increases the output range of the offset map, makes the output boundary of the reachable space smoother, and improves the robustness of the neural network model for obtaining the boundary of the reachable space.

[0116] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0117] Please refer to Figure 6 , which shows a structural block diagram of a reachable space boundary determination device provided by an exemplary embodiment of the present application. The reachable space boundary determination device 600 can be applied to a terminal. The reachable space boundary determination device includes:

[0118] An image acquisition module 601 is used to extract features from the first fisheye image to obtain a feature pyramid, wherein the feature pyramid includes multiple layers of image features of the first fisheye image;

[0119] A boundary acquisition module 602 is used to acquire an initial boundary of a reachable space in the first fisheye image according to image features of a target layer in the feature pyramid and a regression model, wherein the target layer is a layer in the feature pyramid containing the largest number of features, and the regression model is a model trained based on at least two sample layers and acquiring an initial boundary with the same width as the sample layer;

[0120] The offset map acquisition module 603 is used to acquire a target heat map corresponding to the target layer and an offset map corresponding to the target layer according to the target layer and the heat map model, wherein the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image;

[0121] The boundary determination module 604 is used to determine the actual boundary of the reachable space according to the initial boundary and the offset map.

[0122] In summary, the terminal of the present application obtains a feature pyramid by performing feature extraction on the first fisheye image; obtains the initial boundary of the reachable space in the first fisheye image according to the image features of the target layer in the feature pyramid and the regression model; obtains the target heat map corresponding to the target layer and the offset map corresponding to the target layer according to the target layer and the heat map model, the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image; determines the actual boundary of the reachable space according to the initial boundary and the offset map. In the process of performing boundary detection of the reachable space on the first fisheye image of the present application, the offset map obtained by combining the prediction range of the threshold function as an integer makes the prediction range of the offset map not limited by decimals, increases the output range of the offset map, makes the output boundary of the reachable space smoother, and improves the robustness of the neural network model for obtaining the boundary of the reachable space.

[0123] Optionally, the offset map acquisition module 603 includes: a first acquisition unit, a second acquisition unit and a third acquisition unit;

[0124] The first acquisition unit is used to extract the thermal features of the target layer through the thermal map model to obtain a thermal feature map, wherein the image parameters of the thermal feature map are the same as those of the target layer, and the image parameters include width, length, and number of channels of pixel points;

[0125] The second acquisition unit is used to acquire the thermal value of the thermal feature map through the thermal map model to obtain the target thermal map, wherein the target thermal map is a single-channel image, and the image size of the target thermal map is the same as the image size of the target layer;

[0126] The third acquisition unit is used to obtain an offset value for the thermal feature map through the thermal map model to obtain the offset map, where the offset map is a single-channel image, and the image size of the offset map is the same as the image size of the target layer.

[0127] Optionally, the device further comprises:

[0128] A first determination module is used to determine a first pixel point corresponding to the initial boundary in the offset map after the thermal characteristic map is convolved through the third convolution layer to obtain the offset map;

[0129] A pixel expansion module, used to expand the pixel points included in the initial boundary in the offset image according to a preset expansion rule to obtain an expanded image area, wherein the preset expansion rule includes expanding N columns to both sides with the pixel point corresponding to the initial boundary as the center, where N is an integer greater than 0;

[0130] The boundary determination module 604 is further configured to determine an actual boundary of the reachable space according to the initial boundary and the expanded image region in the offset map.

[0131] Optionally, the boundary determination module 604 is further configured to determine each pixel point included in the expanded image area in the offset map;

[0132] The initial boundary and the pixel values ​​of each pixel point included in the expanded image area in the offset map are summed to determine the actual boundary of the reachable space.

[0133] Optionally, the device further comprises:

[0134] A second determination module, configured to determine, in the target thermal map, a second pixel point corresponding to the initial boundary before determining the actual boundary of the reachable space according to the initial boundary and the offset map;

[0135] A maximum value acquisition module, used for acquiring a local maximum value of a pixel value corresponding to each pixel point within a preset image range of each second pixel point in the target thermal map;

[0136] A first updating module, used for updating the initial boundary according to the positions of the pixels of each local maximum value in the first fisheye image;

[0137] The boundary determination module 604 is further configured to determine an actual boundary of the reachable space according to the updated initial boundary and the offset map.

[0138] Optionally, the boundary acquisition module 602 includes: a fourth acquisition unit and a boundary determination unit;

[0139] The fourth acquisition unit is used to extract the boundary of the target layer according to the regression model to obtain the boundary of the reachable space in the target layer;

[0140] The boundary determination unit is used to use the boundary of the reachable space in the target layer as the initial boundary of the reachable space in the first fisheye image, the initial boundary is a single-channel array, and the width of the initial boundary is the same as the width of the target layer.

[0141] Optionally, the image acquisition module includes: a fifth acquisition unit and a pyramid generation unit;

[0142] The fifth acquisition unit is used to acquire a multi-layer feature image according to the first fisheye image and the target residual network;

[0143] The pyramid generating unit is used to generate the feature pyramid according to the multi-layer feature image and the pyramid network.

[0144] Figure 7 is a schematic diagram of the structure of a terminal provided by an exemplary embodiment of the present application. Figure 7 As shown, the terminal 700 includes a central processing unit (CPU) 701, a system memory 704 including a random access memory (RAM) 702 and a read only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701. The terminal 700 also includes a basic input / output system (I / O system) 708 for helping to transmit information between various components in the computer, and a large-capacity storage device 707 for storing an operating system 712, application programs 713 and other program modules 714.

[0145] The basic input / output system 706 includes a display 708 for displaying information and an input device 709 such as a mouse and a keyboard for user inputting information. The display 708 and the input device 709 are connected to the central processing unit 701 through an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include an input / output controller 710 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, a printer, or other types of output devices.

[0146] The mass storage device 707 is connected to the central processing unit 701 via a mass storage controller (not shown) connected to the system bus 705. The mass storage device 707 and its associated computer readable medium provide non-volatile storage for the terminal 700. That is, the mass storage device 707 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0147] The computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage media is not limited to the above. The above-mentioned system memory 704 and mass storage device 707 can be collectively referred to as memory.

[0148] The terminal 700 can be connected to the Internet or other network devices through a network interface unit 711 connected to the system bus 705 .

[0149] The memory also includes one or more programs, which are stored in the memory. The central processing unit 701 implements all or part of the steps performed by the terminal in the methods provided in the above embodiments of the present application by executing the one or more programs. Optionally, the above terminal can be integrated in a vehicle as a vehicle-mounted terminal.

[0150] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0151] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a high-density digital video disc (Digital Video Disc, DVD)), or a semiconductor medium (e.g., a solid-state drive (Solid State Disk, SSD)), etc.

[0152] The embodiment of the present application also discloses a vehicle, which includes an on-board terminal, and the on-board terminal includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the boundary determination method of the reachable space in the above method embodiment. Optionally, the above terminal can be the on-board terminal in this embodiment.

[0153] The embodiment of the present application further discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the method in the above method embodiment when executed by a processor.

[0154] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0155] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0156] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0159] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0160] The above is an example introduction to a method, device, terminal and storage medium for determining the boundary of a reachable space disclosed in an embodiment of the present application. Individual examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for determining the boundary of a reachable space, characterized in that: The method is executed by a terminal, and includes: Performing feature extraction on the first fisheye image to obtain a feature pyramid, wherein the feature pyramid includes multiple layers of image features of the first fisheye image; According to the image features of the target layer in the feature pyramid and the regression model, an initial boundary of the reachable space in the first fisheye image is obtained, wherein the target layer is a layer in the feature pyramid containing the largest number of features, and the regression model is a model trained based on at least two sample layers and obtaining an initial boundary with the same width as the sample layer; the at least two sample layers are obtained by extracting features from different fisheye images to obtain a feature pyramid, and each sample layer is obtained by annotating the feature pyramid; According to the target layer and the heat map model, a target heat map corresponding to the target layer and an offset map corresponding to the target layer are obtained, wherein the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image; An actual boundary of the reachable space is determined according to the initial boundary and the offset map.

2. The method according to claim 1, characterized in that The step of obtaining a target thermal map corresponding to the target layer and an offset map corresponding to the target layer according to the target layer and the thermal map model includes: Extracting thermal features of the target layer through the thermal map model to obtain a thermal feature map, wherein image parameters of the thermal feature map are the same as those of the target layer, and the image parameters include width, length, and number of channels of pixel points; Obtaining a thermal value from the thermal feature map through the thermal map model to obtain the target thermal map, wherein the target thermal map is a single-channel image, and the image size of the target thermal map is the same as the image size of the target layer; The offset value of the thermal feature map is obtained through the thermal map model to obtain the offset map, which is a single-channel image. The image size of the offset map is the same as the image size of the target layer.

3. The method according to claim 2, characterized in that After obtaining the offset value of the thermal characteristic map to obtain the offset map, the method further includes: In the offset map, determining a first pixel point corresponding to the initial boundary; Expanding the pixel points included in the initial boundary in the offset image according to a preset expansion rule to obtain an expanded image area, wherein the preset expansion rule includes expanding N columns to both sides with the pixel point corresponding to the initial boundary as the center, where N is an integer greater than 0; The determining, according to the initial boundary and the offset map, an actual boundary of the reachable space includes: An actual boundary of the reachable space is determined according to the initial boundary and the expanded image region in the offset map.

4. The method according to claim 3, characterized in that The determining, according to the initial boundary and the expanded image area in the offset map, an actual boundary of the reachable space comprises: Determine each pixel point included in the expanded image area in the offset map; The initial boundary and the pixel values ​​of each pixel point included in the expanded image area in the offset map are summed to determine the actual boundary of the reachable space.

5. The method according to any one of claims 1 to 4, characterized in that: Before determining the actual boundary of the reachable space according to the initial boundary and the offset map, the method further includes: In the target heat map, determining a second pixel point corresponding to the initial boundary; Obtaining the local maximum value of the pixel value corresponding to each pixel point within a preset image range of each second pixel point in the target heat map; updating the initial boundary according to the positions of the pixels of each local maximum value in the first fisheye image; The determining, according to the initial boundary and the offset map, an actual boundary of the reachable space includes: The actual boundary of the reachable space is determined according to the updated initial boundary and the offset map.

6. The method according to claim 1, characterized in that The obtaining, according to the target layer in the feature pyramid and the regression model, an initial boundary of a reachable space in the first fisheye image includes: Extracting the boundary of the target layer according to the regression model to obtain the boundary of the reachable space in the target layer; The boundary of the reachable space in the target layer is used as the initial boundary of the reachable space in the first fisheye image, the initial boundary is a single-channel array, and the width of the initial boundary is the same as the width of the target layer.

7. The method according to claim 1, characterized in that The step of extracting features from the first fisheye image to obtain a feature pyramid includes: Acquire a multi-layer feature image according to the first fisheye image and the target residual network; The feature pyramid is generated according to the multi-layer feature image and the pyramid network.

8. A device for determining the boundary of a reachable space, characterized in that: The device is applied to a terminal, and the device includes: An image acquisition module, used for performing feature extraction on the first fisheye image to obtain a feature pyramid, wherein the feature pyramid includes multiple layers of image features of the first fisheye image; A boundary acquisition module, used to acquire the initial boundary of the reachable space in the first fisheye image according to the image features of the target layer in the feature pyramid and the regression model, wherein the target layer is the layer containing the largest number of features in the feature pyramid, and the regression model is a model trained based on at least two sample layers and acquiring an initial boundary with the same width as the sample layer; the at least two sample layers are obtained by extracting features from different fisheye images to obtain a feature pyramid, and each sample layer is obtained by annotating the feature pyramid; An offset map acquisition module is used to acquire a target heat map corresponding to the target layer and an offset map corresponding to the target layer according to the target layer and the heat map model, wherein the prediction range of the threshold function for generating the offset map in the heat map model is an integer, and the pixel value of the pixel point in the offset map is used to indicate the offset value of the corresponding position of the pixel point in the first fisheye image; A boundary determination module is used to determine the actual boundary of the reachable space according to the initial boundary and the offset map.

9. A terminal, characterized in that: The terminal includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method for determining the boundary of a reachable space as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining the boundary of a reachable space as described in any one of claims 1 to 7 is implemented.

Images