Close-range free space detection method and device based on convolution enhancement, and medium
The location replacement and weight of the data of the semantic segmentation and characteristics of the characteristic chart solve the problem of poor detection effect of the fish eye camera at the parking scenario scene, and realize the efficient testing of the proximal free space.
Patent Information
- Application Number
- CN202411393192.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-01-28
AI Technical Summary
The prior art has poor near-end detection effect in free space in parking scenes, mainly due to the severe distortion of the fisheye camera and the poor effect of training models on close-range images.
By obtaining fish -eye images for semantic segmentation, separation of proximal and remote feature diagrams, and strengthening the weight through position replacement and weight, new feature map data is generated for feature extraction to improve the effect of proximal detection.
By enhanced the near -end feature extraction, the accuracy and effect of proximal detection significantly improved the problems of poor short -end detection of fish -eye image image.
Smart Images
Figure CN119379965A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of software development, and in particular to a close-range free-space detection method, device and medium based on convolution enhancement. Background Art
[0002] Currently, semantic segmentation and target detection are mainly used to detect drivable areas. However, the free space in parking scenarios requires very high detection results at the near end, and current technology does not have a solution optimized for near-end detection.
[0003] The near-end detection is not good because: the closer to the fisheye camera, the more serious the distortion, which affects the recognition effect; the fisheye camera has a low field of view and the collected images have a narrow field of view; there are relatively few close-range images in the target detection training set, and the trained model has poor recognition effect. Summary of the invention
[0004] In response to the above technical problems, the present invention provides a close-range free-space detection method, device, and medium based on convolution enhancement, which can improve the effect of near-end detection.
[0005] A first aspect of the present invention provides a close-range free space detection method based on convolution enhancement, comprising: Acquire a fisheye image, perform semantic segmentation processing on the fisheye image, obtain feature map data and mask data, and divide the mask data into proximal feature map data and distal feature map data; Replace the positions of the near-end feature map data and the far-end feature map data to obtain new feature map data, perform feature extraction on the new feature map data, and obtain map data based on the feature extraction result; Close range free space detection is performed based on the map data.
[0006] In an optional implementation, the performing feature extraction on the new feature graph data and obtaining graph data based on the feature extraction result includes: Perform feature extraction on the new feature graph data to obtain first feature data, add the first feature data to the feature graph data to obtain second feature data; and obtain graph data according to the second feature data.
[0007] In an optional implementation, the extracting features from the new feature map data to obtain first feature data, and adding the first feature data to the feature map data to obtain second feature data, includes: The matrix of the first feature data is added to the matrix of the feature graph data to obtain a matrix of the second feature data.
[0008] In an optional implementation manner, the step of replacing the position of the proximal feature map data with the distal feature map data to obtain new feature map data includes: The feature map data is multiplied by the mask data to obtain the new feature map data; wherein the mask data is marked as upper far-end feature map data and lower near-end feature map data; and the near-end feature map data and the far-end feature map data have different weights.
[0009] In an optional embodiment, the weight of the proximal feature map data is greater than the weight of the distal feature map data.
[0010] In an optional implementation, the acquiring of the fisheye image, performing semantic segmentation processing on the fisheye image, and obtaining feature map data and mask data includes: The fisheye image is semantically segmented to obtain feature map data, the feature map data is divided into proximal feature map data and distal feature map data, and the proximal feature map data is weighted.
[0011] A second aspect of the present invention provides a close-range free-space detection device based on convolution enhancement, comprising: A first processing module is used to acquire a fisheye image, perform semantic segmentation processing on the fisheye image, obtain feature map data and mask data, and divide the mask data into proximal feature map data and distal feature map data; A second processing module is used to replace the position of the near-end feature map data with the far-end feature map data to obtain new feature map data, perform feature extraction on the new feature map data, and obtain map data based on the feature extraction result; A detection module performs close-range free space detection based on the image data.
[0012] In an optional embodiment, the second processing module also includes a processing submodule, and the processing submodule is used to extract features from the new feature map data to obtain first feature data, add the first feature data to the feature map data to obtain second feature data; and obtain map data based on the second feature data.
[0013] According to a third aspect of the present invention, there is provided an electronic device, comprising: At least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the close-range free space detection method based on convolution enhancement as described in the first aspect of the embodiment of the present invention.
[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the close-range free space detection method based on convolution enhancement as described in the first aspect of the embodiment of the present invention is executed.
[0015] The present invention divides mask data into near-end feature map data and far-end feature map data, and obtains new feature map data by replacing the positions of the near-end feature map data and the far-end feature map data, so that the model pays more attention to the feature extraction of the near-end feature map data, thereby improving the effect of near-end detection and overcoming the problem of poor near-end detection of fisheye images. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0017] Figure 1 The figure is a flow chart of a close-range free-space detection method based on convolution enhancement in an embodiment of the present invention.
[0018] Figure 2 It is a schematic diagram of creating mask data based on feature map data in an embodiment of the present invention.
[0019] Figure 3 Schematic diagram of feature learning of image features by a semantic segmentation model in the prior art.
[0020] Figure 4 It is a schematic diagram of feature learning of image features by a semantic segmentation model in an embodiment of the present invention.
[0021] Figure 5 The figure is a flow chart of another close-range free-space detection method based on convolution enhancement in an embodiment of the present invention.
[0022] Figure 6 This is a rendering of the recognition effect using the existing technology. Figure 6 a is a parking space image identified in the prior art, Figure 6 b is another parking space image identified in the prior art.
[0023] Figure 7 This is a diagram showing the effect of using the method provided by the present invention. Figure 7 a corresponds to Figure 6 The effect diagram of a, Figure 7 b is the corresponding Figure 6 Effect diagram of b.
[0024] Figure 8 The module diagram of a close-range free-space detection device based on convolution enhancement in an embodiment of the present invention.
[0025] Fig. 9 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0027] It should be understood that the terms "first", "second", "third", etc. in the claims, specifications and drawings of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.
[0028] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the claims, the singular forms of "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in the specification of the present invention and the claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.
[0029] See also Figure 1 The present invention provides a close-range free-space detection method based on convolution enhancement, comprising the following steps.
[0030] S100: Acquire a fisheye image, perform semantic segmentation processing on the fisheye image, obtain feature map data and mask data, and divide the mask data into proximal feature map data and distal feature map data.
[0031] The semantic segmentation model obtains the fisheye image, uses the convolution function to extract the features of the fisheye image to obtain the feature map data, and then creates a mask to obtain the mask data (mask). Due to the structure of the fisheye image, the distortion at the far end is smaller than that at the near end. The semantic segmentation model has better detection effect on the far end than on the near end. Therefore, the near end target can be replaced with the far end, and the semantic segmentation model can be used to perform semantic segmentation and subsequent target recognition, and then returned to the near end.
[0032] In this step, the mask data can be divided into two parts, one part is the near-end feature map data, and the other part is the far-end feature map data. Figure 2 As shown, the height h direction of the feature map is divided into two parts, for example, the far-end feature map data in the upper part and the near-end feature map data in the lower part.
[0033] In some other embodiments of the present invention, Figure 2 The width w of the feature map is divided into two parts, for example, the far feature map data of the left half and the near feature map data of the right half. Since the fisheye image is roughly circular, the processing of the fisheye image is not necessarily limited to the upper and lower parts. The image can be rotated to obtain a similar processing process. For example, the two parts divided in the width w direction are rotated clockwise to obtain two parts divided in the height h direction.
[0034] S200: Replace the positions of the proximal feature map data and the distal feature map data to obtain new feature map data, perform feature extraction on the new feature map data, and obtain map data based on the feature extraction result.
[0035] Since the far-end feature map data is not severely distorted and has a good recognition effect, good recognition performance can still be achieved after the position change. In this step, after the near-end feature map data is replaced and changed to the far-end, the semantic segmentation model pays more attention to the extraction of the near-end feature map data, and the recognition effect is better.
[0036] In some embodiments, different weight values can be assigned to the upper and lower parts of the above-mentioned feature map. For example, the weight value of the distal feature map data in the upper part is set to 1, and the weight of the proximal feature map data in the lower part is set to 2, so as to strengthen the weight of the proximal feature map data and achieve the effect of convolution enhancement during feature extraction.
[0037] After position replacement, new feature map data is obtained, and feature learning is performed on the new feature map data using the same convolution function as that used to obtain the above feature map data, and the map data is obtained after feature extraction.
[0038] S300: Perform close-range free space detection based on the image data.
[0039] In this step, target recognition and free space detection can be performed based on the image data. The free space detection is the detection of the vehicle's drivable area, which involves obstacle recognition, target recognition (lane line recognition), etc. in the fisheye image. Close-range free space detection or target recognition or obstacle recognition based on feature data belongs to the prior art and will not be described in detail. It should be understood that the present invention can also be used in other scenarios that require proximal detection, and the detection of the drivable area is only one of its application scenarios.
[0040] Furthermore, gradient vanishing means that when training a deep neural network, as the error gradient is passed back from the output layer to the input layer layer by layer, the gradient may gradually decrease due to the multiplication effect, and eventually become very close to zero, causing the previous hidden layer to learn slowly or even stop updating, thereby affecting the performance of the entire network. This problem mainly occurs when using certain specific activation functions (such as Sigmoid or Tanh) or when the initialization weights are too large or too small. In some embodiments, to prevent the gradient from vanishing, the new feature map data after the near-end and far-end replacement is added to the feature map data to obtain the final map data.
[0041] For example, feature extraction is performed on the new feature graph data to obtain first feature data, the first feature data is added to the feature graph data to obtain second feature data; and graph data is obtained according to the second feature data.
[0042] For example, a convolution 3×3 alignment is used to perform feature learning on the new feature map data, and the new feature map data is extracted to obtain the first feature data, Featuremap_new=conv3×3(Featuremap_new), and the above feature map data is also obtained by feature learning using a convolution 3×3 alignment: Featuremap =conv3×3(Featuremap). The matrix of the first feature data and the matrix of the feature map data can be added to obtain the matrix of the second feature data.
[0043] Therefore, the feature map data Featuremap matrix is 4×64×120×160, and the new feature map data Featuremap_new matrix is 4×64×120×160. Then the graph data obtained after addition is: Featuremap_result = torch.cat(Featuremap, Featuremap_new), and the Featuremap_result matrix is 4×128×120×160.
[0044] The present invention recognizes fisheye images through a semantic segmentation model, extracts feature map data after feature learning, and creates mask data based on the input feature map data. Then, the near-end feature map data of the mask data is replaced with the far-end feature map data to obtain new feature map data, so that the model pays more attention to the feature extraction of the near-end feature map data, thereby improving the effect of near-end detection and overcoming the problem of poor near-end detection of fisheye images.
[0045] The present invention will be further described below. Figure 3 As shown, Figure 3 In the prior art, the semantic segmentation model learns the features of the image, and only uses the convolutional layer Conv to directly identify the feature map data Featuremap. Figure 4 The present invention utilizes the convolution layer Conv to identify the feature map data, and then creates mask data according to the feature map data, divides the mask data into the proximal feature map data and the distal feature map data, and then replaces the proximal feature map data with the distal feature map data to obtain new feature map data, and finally fuses the feature map data with the new feature map data, and then performs function analysis cat or feature fusion, and finally performs image recognition that serves the detection purpose, such as close-range free space detection in a drivable area.
[0046] See also Figure 5 The close-range free-space detection method based on convolution enhancement includes the following steps.
[0047] S510: Acquire a fisheye image, perform semantic segmentation processing on the fisheye image, and obtain feature map data and mask data.
[0048] The semantic segmentation model extracts feature map data of the fisheye image and creates mask data based on the feature map data.
[0049] S520: Divide the mask data into proximal feature map data and distal feature map data, and perform weight enhancement on the proximal feature map data.
[0050] The mask data is marked as the upper far-end feature map data and the lower near-end feature map data. Figure 2 The weight of the proximal feature map data is greater than the weight of the distal feature map data. The weights of the proximal feature map data and the distal feature map data are different, so that the semantic segmentation model can pay more attention to the extraction of the proximal feature map data.
[0051] S530: Multiply the feature map data and the mask data, and replace the positions of the near-end feature map data and the far-end feature map data to obtain new feature map data.
[0052] New feature map data featuremap_new = featuremap (feature map data) × mask (mask data).
[0053] S540: extract features from the new feature graph data to obtain first feature data, add the first feature data to the feature graph data to obtain second feature data; and obtain graph data according to the second feature data.
[0054] A convolution 3×3 alignment is used to perform feature extraction on the new feature map data to obtain the first feature data. The above feature map data is also data obtained by a convolution 3×3 alignment.
[0055] The matrix of the first feature data is added to the matrix of the feature map data to obtain the matrix of the second feature data. For example, the matrix of the feature map data Featuremap is 4×64×120×160, and the matrix of the new feature map data Featuremap_new is 4×64×120×160. Then the graph data obtained after the addition is: Featuremap_result = torch.cat(Featuremap, Featuremap_new), and the Featuremap_result matrix is 4×128×120×160.
[0056] S550: Perform close-range free space detection based on the image data.
[0057] See also Figure 6 and Figure 7 , Figure 6 This is an effect diagram identified using existing technology. Figure 6 a is a parking space image identified in the prior art, Figure 6 b is another parking space image identified in the prior art. Figure 7 This is a diagram showing the effect of identification using the method provided by the present invention. Figure 7 a corresponds to Figure 6 The effect diagram of a, Figure 7 b is the corresponding Figure 6 b. Comparison Figure 6 a and Figure 7 a, Figure 6 a Only part of the flower field edge and the vehicle at the far end were identified; Figure 7 a identifies the entire edge of the flower field (including the proximal edge of the flower field) and the vehicle. Figure 6 b only recognizes the far wall and vehicle, Figure 7b recognizes the far wall and vehicle, and the near cone. It can be seen that the present invention is superior to the prior art in the near-end recognition. It should be understood that the data marked in the figure are used to characterize different object types and are not intended to limit the present invention.
[0058] See also Figure 8 The present invention also provides a close-range free-space detection device based on convolution enhancement, comprising: a first processing module 81, a second processing module 82, and a detection module 83.
[0059] The first processing module 81 is used to acquire a fisheye image, perform semantic segmentation processing on the fisheye image, obtain feature map data and mask data, and divide the mask data into proximal feature map data and distal feature map data.
[0060] For example, the feature map data is multiplied by the mask data to obtain the new feature map data; wherein the mask data is marked as the upper part of the far-end feature map data and the lower part of the near-end feature map data; and the weights of the near-end feature map data and the far-end feature map data are different.
[0061] The fisheye image is semantically segmented to obtain feature map data, the feature map data is divided into near-end feature map data and far-end feature map data, and the near-end feature map data is weighted, wherein the weight of the near-end feature map data is greater than the weight of the far-end feature map data.
[0062] The second processing module 82 is used to replace the position of the near-end feature map data with the far-end feature map data to obtain new feature map data, perform feature extraction on the new feature map data, and obtain map data based on the feature extraction result.
[0063] The detection module 83 performs close-range free space detection based on the image data.
[0064] Furthermore, the second processing module 83 also includes a processing submodule, which is used to extract features from the new feature graph data to obtain first feature data, add the first feature data to the feature graph data to obtain second feature data, and obtain graph data according to the second feature data. For example, the matrix of the first feature data is added to the matrix of the feature graph data to obtain the matrix of the second feature data.
[0065] For the description of the short-range free-space detection device based on convolution enhancement, reference may be made to the short-range free-space detection method based on convolution enhancement, which will not be described in detail.
[0066] like Fig. 9 As shown, the present invention also provides an electronic device, including: At least one processor; and at least one memory in communication with the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned close-range free space detection method based on convolution enhancement.
[0067] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned close-range free space detection method based on convolution enhancement is implemented.
[0068] It can be understood that computer-readable storage media may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. A computer program includes a computer program code. The computer program code may be in source code form, object code form, an executable file, or some intermediate form, etc. A computer-readable storage medium may include: any entity or device capable of carrying a computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.
[0069] In certain embodiments of the present invention, the electronic device may include a controller or a processor, and the controller is a single-chip microcomputer chip that integrates a processor, a memory, a communication module, etc. The processor may refer to a processor included in the controller. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
[0070] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.
[0071] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A close-range free-space detection method based on convolution enhancement, characterized in that: include: Acquire a fisheye image, perform semantic segmentation processing on the fisheye image, obtain feature map data and mask data, and divide the mask data into proximal feature map data and distal feature map data; Replace the positions of the near-end feature map data and the far-end feature map data to obtain new feature map data, perform feature extraction on the new feature map data, and obtain map data based on the feature extraction result; Close range free space detection is performed based on the map data.
2. The close-range free-space detection method based on convolution enhancement according to claim 1, characterized in that: The step of extracting features from the new feature graph data and obtaining graph data based on the feature extraction result includes: Perform feature extraction on the new feature graph data to obtain first feature data, add the first feature data to the feature graph data to obtain second feature data; and obtain graph data according to the second feature data.
3. The close-range free-space detection method based on convolution enhancement according to claim 2, characterized in that: The extracting features from the new feature graph data to obtain first feature data, and adding the first feature data to the feature graph data to obtain second feature data, comprises: The matrix of the first feature data is added to the matrix of the feature graph data to obtain a matrix of the second feature data.
4. The close-range free-space detection method based on convolution enhancement according to claim 1 or 2, characterized in that: The step of replacing the position of the proximal feature map data with the distal feature map data to obtain new feature map data includes: The feature map data is multiplied by the mask data to obtain the new feature map data; wherein the mask data is marked as upper far-end feature map data and lower near-end feature map data; and the near-end feature map data and the far-end feature map data have different weights.
5. The close-range free-space detection method based on convolution enhancement according to claim 4, characterized in that: The weight of the proximal feature map data is greater than the weight of the distal feature map data.
6. The close-range free-space detection method based on convolution enhancement according to claim 1, characterized in that: The step of acquiring a fisheye image and performing semantic segmentation processing on the fisheye image to obtain feature map data and mask data includes: The fisheye image is semantically segmented to obtain feature map data, the feature map data is divided into proximal feature map data and distal feature map data, and the proximal feature map data is weighted.
7. A short-range free-space detection device based on convolution enhancement, characterized in that: include: A first processing module is used to acquire a fisheye image, perform semantic segmentation processing on the fisheye image, obtain feature map data and mask data, and divide the mask data into proximal feature map data and distal feature map data; A second processing module is used to replace the position of the near-end feature map data with the far-end feature map data to obtain new feature map data, perform feature extraction on the new feature map data, and obtain map data based on the feature extraction result; A detection module performs close-range free space detection based on the image data.
8. The close-range free-space detection device based on convolution enhancement according to claim 7, characterized in that: The second processing module also includes a processing submodule, which is used to extract features from the new feature map data to obtain first feature data, add the first feature data to the feature map data to obtain second feature data; and obtain map data based on the second feature data.
9. An electronic device, characterized in that: include: at least one processor; And at least one memory communicatively connected to the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the close-range free space detection method based on convolution enhancement as described in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the close-range free-space detection method based on convolution enhancement as claimed in any one of claims 1 to 6 is executed.