Obstacle detection method, device, equipment and medium

Abstract

The invention discloses an obstacle detection method, and the method comprises the steps: obtaining a first image which is an image coded based on a three-primary-color light mode; reconstructing the first image to obtain a second image, wherein the second image is a hyperspectral image; hyperspectral features are extracted from the hyperspectral image, candidate objects in the hyperspectral image are classified according to the hyperspectral features, and an obstacle detection result is obtained. Due to the fact that the hyperspectral features corresponding to different materials are different, objects with similar colors but different materials can be distinguished by classifying the candidate objects in the hyperspectral image based on the hyperspectral features, on the basis, detection of obstacles with the same or similar colors as the environment color can be achieved, and the omission ratio is reduced. Besides, according to the method, the hyperspectral image can be obtained through an image reconstruction mode by adopting a common camera, and an imaging spectrometer does not need to be adopted, so that the cost is relatively low, and the feasibility is relatively high. The invention further discloses a corresponding device, equipment and a medium.

Description

[0001] This application is a divisional application. The application number of the original application is 201910954529.X, and the original application date is October 9, 2019. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of communications, and in particular to an obstacle detection method, apparatus, device, medium and computer program product. Background Art

[0003] With the rapid development of artificial intelligence, assisted driving and autonomous driving technologies have emerged. When assisted driving or autonomous driving functions are enabled, it is necessary to perceive the surrounding environment, that is, to perceive information such as pedestrians, vehicles, lane lines, drivable areas, and obstacles on the driving path to avoid collisions with other vehicles, pedestrians, obstacles, or deviating from lane lines, etc.

[0004] For the perception of obstacles, the industry has provided an obstacle detection method based on a binocular camera. The binocular camera can realize the disparity detection of the image, so that the disparity of the obstacle can be obtained, and the obstacle detection can be realized based on the disparity.

[0005] However, binocular cameras have problems such as high baseline requirements and high calibration requirements. In addition, when the color of the obstacle is similar to the color of the environment, it is very likely to cause missed detection, resulting in the inability of the binocular camera-based detection system to detect it, posing certain safety hazards to driving. Summary of the invention

[0006] In view of this, the present application provides an obstacle detection method, which reconstructs an image encoded based on the three primary color light pattern into a hyperspectral image. Based on the hyperspectral image, obstacle detection based on material can be achieved, solving the problem of missed detection caused by the similar color of the obstacle and the environment. It is also low-cost and highly feasible.

[0007] A first aspect of an embodiment of the present application provides an obstacle detection method, the method comprising:

[0008] Acquire a first image, where the first image is an image encoded based on a three-primary color light mode;

[0009] Reconstructing the first image to obtain a second image, where the second image is a hyperspectral image;

[0010] A hyperspectral feature is extracted from the hyperspectral image, and candidate objects in the hyperspectral image are classified according to the hyperspectral feature to obtain an obstacle detection result.

[0011] Optionally, reconstructing the first image to obtain the second image includes:

[0012] extracting spatial features of the first image;

[0013] According to the spatial features of the first image, image reconstruction is performed using the correspondence between the spatial features and the spectral features to obtain a second image.

[0014] Optionally, the method further includes:

[0015] Obtaining a data dictionary from a configuration file, wherein the data dictionary includes a correspondence between spatial features and spectral features; or,

[0016] Sample data is obtained, and machine learning is performed using the sample data to obtain a correspondence between spatial features and spectral features.

[0017] Optionally, the method further includes:

[0018] fusing the hyperspectral feature and the spatial feature of the first image to obtain a fused feature;

[0019] Then, classifying the candidate objects in the hyperspectral image according to the hyperspectral features includes:

[0020] The candidate objects in the hyperspectral image are classified according to the fused features.

[0021] Optionally, the hyperspectral features and the spatial features of the first image are fused by a Bayesian data fusion algorithm.

[0022] Optionally, the first image includes an RGB image, an RCCC image, an RCCB image or an RGGB image.

[0023] Optionally, the obstacle detection result includes the location and material of the obstacle;

[0024] The method further comprises:

[0025] Determine a drivable area according to the location and material of the obstacle;

[0026] The drivable area is sent to a vehicle controller to instruct the vehicle to travel according to the drivable area.

[0027] A second aspect of an embodiment of the present application provides an obstacle detection device, the device comprising:

[0028] An acquisition module, used for acquiring a first image, where the first image is an image encoded based on a three-primary color light mode;

[0029] A reconstruction module, used for reconstructing the first image to obtain a second image, where the second image is a hyperspectral image;

[0030] The detection module is used to extract hyperspectral features from the hyperspectral image, classify candidate objects in the hyperspectral image according to the hyperspectral features, and obtain obstacle detection results.

[0031] Optionally, the reconstruction module is specifically used for:

[0032] extracting spatial features of the first image;

[0033] According to the spatial features of the first image, image reconstruction is performed using the correspondence between the spatial features and the spectral features to obtain a second image.

[0034] Optionally, the acquisition module is further used for:

[0035] Obtaining a data dictionary from a configuration file, wherein the data dictionary includes a correspondence between spatial features and spectral features; or,

[0036] Sample data is obtained, and machine learning is performed using the sample data to obtain a correspondence between spatial features and spectral features.

[0037] Optionally, the device further comprises:

[0038] A fusion module, used for fusing the hyperspectral feature and the spatial feature of the first image to obtain a fusion feature;

[0039] The detection module is specifically used for:

[0040] The candidate objects in the hyperspectral image are classified according to the fused features.

[0041] Optionally, the fusion module is specifically used for:

[0042] The hyperspectral features and the spatial features of the first image are fused by a Bayesian data fusion algorithm.

[0043] Optionally, the first image includes an RGB image, an RCCC image, an RCCB image or an RGGB image.

[0044] Optionally, the obstacle detection result includes the location and material of the obstacle;

[0045] The device also includes:

[0046] A determination module, used to determine a drivable area according to the location and material of the obstacle;

[0047] The sending module is used to send the drivable area to the vehicle controller to instruct the vehicle to travel according to the drivable area.

[0048] A third aspect of an embodiment of the present application provides a driving assistance system, including a processor and a memory:

[0049] The memory is used to store computer programs;

[0050] The processor is used to execute the obstacle detection method described in the first aspect according to the instructions in the computer program.

[0051] A fourth aspect of the embodiments of the present application provides a vehicle, the vehicle comprising the driving assistance system and the controller according to the third aspect;

[0052] The controller is used to control the vehicle driving according to the obstacle detection result output by the driving assistance system.

[0053] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the obstacle detection method described in the first aspect of the present application.

[0054] A sixth aspect of the embodiments of the present application provides a computer program product comprising computer-readable instructions. When the computer-readable instructions are executed on a computer, the computer executes the obstacle detection methods described in the above aspects.

[0055] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0056] The embodiment of the present application provides an obstacle detection method, in which a hyperspectral image is obtained by reconstructing a first image encoded based on a three-primary color light pattern, and a hyperspectral feature is extracted from the hyperspectral image. Since the hyperspectral features corresponding to different materials are different, the candidate objects in the hyperspectral image are classified based on the hyperspectral features, and objects with similar colors but different materials can be distinguished. Based on this, obstacle detection with the same or similar color as the environment can be achieved, and the missed detection rate is reduced. In addition, the method can obtain a hyperspectral image by reconstructing the image using an ordinary camera, without the need for an imaging spectrometer, so the cost is low and it has high feasibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 This is a system architecture diagram of the obstacle detection method in the embodiment of the present application;

[0059] Figure 2 This is a flow chart of an obstacle detection method in an embodiment of the present application;

[0060] Figure 3 This is a flow chart of an obstacle detection method in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of extracting a spectral curve from a hyperspectral image in an embodiment of the present application;

[0062] Figure 5 is an interactive flow chart of the obstacle detection method in an embodiment of the present application;

[0063] Figure 6 is an interactive flow chart of the obstacle detection method in an embodiment of the present application;

[0064] Figure 7 This is a schematic diagram of the structure of an obstacle detection device in an embodiment of the present application;

[0065] Figure 8 A structural diagram of a server in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application provide an obstacle detection method for solving the problems of high baseline requirements and high calibration requirements when a binocular camera performs obstacle detection, as well as the problem of missed detection when the color of the obstacle is similar to the color of the environment, without incurring additional costs.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] It can be understood that the obstacle detection method provided in the embodiment of the present application can be applied to scenes such as autonomous driving (AD) or assisted driving. Taking the assisted driving scene as an example, the method can be applied to the Advanced Driver Assistant System (ADAS), through which the ADAS can realize obstacle target detection (OD), road profile detection (RPD), traffic sign recognition (TSR), and further provide intelligent speed limit reminder (ISLI) and other services. In this way, the driving safety hazards caused by human negligence can be avoided through automatic detection, driving safety can be improved, and the driving experience can be improved due to the significant reduction of driver operations.

[0070] In practical application, the above obstacle detection method can be applied to any processing device with image processing capability, which can be a terminal with a central processing unit (CPU) and / or a graphics processing unit, or a server with a CPU and / or a GPU, wherein the terminal can be a personal computing device (PC) or a workstation, etc. The terminal or server implements the obstacle detection method by communicating with the vehicle's driving assistance system, etc. Of course, in some cases, the terminal can also be a vehicle-mounted terminal, such as a vehicle's own driving assistance system, etc., and the driving assistance system can also independently implement the obstacle detection method.

[0071] The obstacle detection method provided in the embodiment of the present application can be stored in a processing device in the form of a computer program, and the processing device implements the obstacle detection method provided in the embodiment of the present application by running the computer program. The computer program can be independent or integrated into other computer programs. Functional modules, plug-ins or applets, etc.

[0072] Next, the application environment of the obstacle detection method provided in the embodiment of the present application is described in detail. The method can be applied to, but is not limited to, Figure 1 In the application environment shown.

[0073] like Figure 1 As shown, the vehicle is deployed with a driving assistance system 101, and the driving assistance system 101 can call the vehicle's front-view camera to capture the vehicle's surrounding environment to obtain a first image, and can also obtain the first image through a test camera, a rear-view camera or a surround-view camera. The first image is specifically an image encoded based on a three-primary color light pattern. Then the driving assistance system 101 can transmit the first image to a server 102 through a network, such as a wireless communication network such as 4G or 5G. The server 102 reconstructs the first image to obtain a second image, which is specifically a hyperspectral image. Then, hyperspectral features are extracted from the hyperspectral image, and candidate objects in the hyperspectral image are classified according to the hyperspectral features to obtain an obstacle detection result.

[0074] In order to make the technical solution of the present application clearer and easier to understand, the obstacle detection method provided in the embodiment of the present application will be described in detail from the perspective of the server with reference to the accompanying drawings.

[0075] See also Figure 2 The flowchart of the obstacle detection method shown in FIG. 1 includes:

[0076] S201: Acquire a first image.

[0077] The first image is an image encoded based on a three-primary color light model, wherein the three-primary color light model (RGB model) is also called an RGB color model or a red, green, and blue color model, which is an additive color model that adds the three primary colors of red, green, and blue in different proportions to produce light of various colors.

[0078] In practical applications, the image encoded based on the RGB model may be an image obtained by filtering using a general color filter array (CFA), that is, an RGB image, or may be an image obtained by filtering using other filters, which may be determined according to actual needs.

[0079] For example, the Bayer Filter CFA is configured with 1 red light, 1 blue light, and 2 green light filters (i.e., 25% Red, 25% Blue, 50% Green), and the human eye is naturally more sensitive to green. The green light transmittance in the Bayer Filter is better than the other two colors. Therefore, compared with the equivalent processing of the three RGB colors, the image restored by this method has lower noise and clearer details in the eyes of the human eye. In applications that require high-definition images, the image obtained based on the Bayer Filter, that is, the Bayer image, can be selected. Among them, the Bayer image can be divided into four Bayer patterns, including BGGR, GBRG, GRBG or RGGB.

[0080] For another example, in vehicle-mounted forward vision applications, the above CFA can adopt a Red-Monochrome (RCCC) configuration, in which the CFA filter structure includes three blank (Clear-C) filters and one red light filter. Compared with the Bayer Filter, which discards 2 / 3 of the light source during the processing, the RCCC CFA signal sensitivity is higher, and the intensity of the red light is sufficient to determine the situation of the car headlights (white) and taillights (red). Based on this, the RCCC image is suitable for low-light environments and is mainly used in situations where red signs are sensitive, such as traffic light detection, car headlight and taillight detection, etc.

[0081] Taking into account that when performing machine analysis on images, it is generally required that the image has good color resolution ability, therefore, the above CFA configuration can also be 50% transparent, with the remaining red and blue light each accounting for 25%. Based on this, the image obtained is an RCCB image, that is, the above first image can also be an RCCB image.

[0082] In some occasions where color object recognition is required, such as when detecting the driver's status, the above-mentioned first image can also be Monochrome, which is 100% transparent and cannot distinguish colors. However, this configuration has the highest low-light sensitivity and thus has a better detection effect.

[0083] For vehicles with automatic driving functions or assisted driving functions, when the above functions are enabled, the above automatic driving system or assisted driving system can call the camera to capture images to obtain a first image, and the server obtains the first image from the above automatic driving system or assisted driving system so as to subsequently realize obstacle detection through image processing technology.

[0084] It should be noted that, when the server acquires the first image, it may automatically acquire the first image periodically, or it may acquire the first image in response to a request message for obstacle detection when the server receives the request message.

[0085] S202: Reconstruct the first image to obtain a second image, where the second image is a hyperspectral image.

[0086] The so-called hyperspectral image refers to a set of spectral images with a spectral resolution in the range of 10I (i.e., 10-2λ), generally including dozens to hundreds of spectral bands. Specifically in this embodiment, the server can reconstruct the second image using the first image captured by an ordinary camera based on the correspondence between the spatial features and the hyperspectral features, thereby achieving the hyperspectral image without using an imaging spectrometer, and without increasing the hardware cost.

[0087] Specifically, the server may first extract the spatial features of the first image through image processing technology, such as a convolutional neural network, and then reconstruct the image according to the spatial features of the first image using the correspondence between the spatial features and the spectral features to obtain the second image.

[0088] The correspondence between spatial features and spectral features can be obtained in many ways. Figure 3 In the first method, the server can generate a data dictionary based on the RGB image and the hyperspectral image in the existing data. The data dictionary includes the correspondence between the spatial features and the spectral features, and the server writes it into the configuration file. In this way, when the server reconstructs the image, it can obtain the data dictionary from the configuration file, and then reconstruct the RGB image to be analyzed (i.e., the first image) based on the correspondence between the spatial features and the spectral features included in the data dictionary to obtain the hyperspectral image (i.e., the second image).

[0089] Next, the process of obtaining the data dictionary is described in detail.

[0090] From the material composition of the image scene, we can see that although the hyperspectral image is composed of two-dimensional images of dozens or hundreds of bands, since the materials in the image scene will not change drastically, each hyperspectral image scene contains no more than 12 kinds of materials. These characteristics of the hyperspectral image determine that it can be sparsely represented by an appropriate dictionary.

[0091] It can be understood that different materials in the hyperspectral image scene have specific spectral reflectance curves. Although the reflectance of the material varies due to different lighting, shooting angles, object geometry, material inhomogeneity, and water content, the curve trend is fixed. Based on this, Figure 4As shown in the figure, a spectral curve of a spectrum can be extracted for each pixel in the hyperspectral image space. These spectral curves are obtained by linearly superimposing the spectral curves of one or several materials. Therefore, taking the spectral reflectance curves of the materials contained in the hyperspectral image scene as dictionary atoms, all pixel points on the hyperspectral image can be sparsely represented by the dictionary.

[0092] In addition, hyperspectral images have similar spatial correlation as grayscale images in the spatial direction, that is, the material composition structures of adjacent pixels in spatial position are similar. Therefore, each spectral band can be regarded as an independent two-dimensional image. If it is overlapped and divided into blocks and then the spatial dictionary is learned, these blocks can be sparsely represented by the obtained dictionary.

[0093] By performing three-dimensional overlapping block division on the hyperspectral image, both the spatial correlation of the image and its inter-spectral correlation are taken into account, so that the learned dictionary is more in line with the structural characteristics of the hyperspectral image.

[0094] In the second implementation, the server may also obtain sample data and use the sample data for machine learning to obtain the correspondence between spatial features and spectral features. The server may use traditional machine learning algorithms such as random forests to obtain the correspondence between spatial features and spectral features, or may use deep learning to obtain the correspondence between spatial features and spectral features.

[0095] The above correspondence is obtained based on deep learning as an example.

[0096] In one example, see Figure 3 The server extracts features based on the RGB image and the hyperspectral image in the existing data to generate sample data, which includes spatial features extracted from the RGB image and spectral features extracted from the hyperspectral image. The server initializes a convolutional neural network model, which takes spatial features as input and spectral features as output, and then inputs the sample data into the convolutional neural network model. The convolutional neural network model can predict the spectral features corresponding to the spatial features, and then calculates the loss function based on the predicted spectral features and the spectral features included in the sample data, and updates the model parameters of the convolutional neural network model based on the loss function.

[0097] Through continuous updating of a large number of samples, when the loss function of the convolutional neural network model tends to converge, or when the loss function of the convolutional neural network model is less than a preset value, iterative training can be stopped, and the convolutional neural network model at this time can be used to extract the correspondence between spatial features and spectral features. Based on this, after extracting the spatial features of the first image, the above-mentioned spatial features are input into the convolutional neural network model to obtain the corresponding spectral features, and the hyperspectral image can be reconstructed based on the spectral features.

[0098] S203: extracting hyperspectral features from the hyperspectral image, and classifying candidate objects in the hyperspectral image according to the hyperspectral features to obtain an obstacle detection result.

[0099] In practical applications, the server may first determine candidate objects based on the hyperspectral image, for example, the candidate objects may be identified by candidate boxes, and then the candidate objects may be classified according to the hyperspectral features extracted from the hyperspectral image to obtain obstacle detection results.

[0100] Since the hyperspectral features corresponding to different materials are different, the classification of candidate objects with the same or similar colors but different materials based on hyperspectral features has higher accuracy. Based on this, obstacle detection has a higher detection rate and can avoid safety hazards caused by missed detection of obstacles with the same or similar colors.

[0101] Furthermore, the server can also classify the candidate objects based on the hyperspectral features and spatial features to further improve the classification accuracy. Specifically, the server can fuse the hyperspectral features and the spatial features of the first image to obtain fused features, and then classify the candidate objects in the hyperspectral image according to the fused features.

[0102] When performing feature fusion, the server may implement it through a fusion algorithm. As an example of the present application, the server may fuse the hyperspectral features and the spatial features of the first image through a Bayesian data fusion algorithm. It should be noted that the Bayesian data fusion algorithm is only a specific example of the present application. In actual applications, the server may also use other fusion algorithms to fuse the hyperspectral features and the spatial features.

[0103] It is understandable that when the server determines that a candidate object is an obstacle based on hyperspectral features, the location and material of the obstacle can be output as obstacle detection results. It should be noted that this application protects the obstacle detection method based on material features, so the interface that describes this information also falls within the scope of protection of this application. Based on this, improvements can also be made to the corresponding interfaces in the relevant standards.

[0104] For example, for the target interface detected in ISO 23150, see Table 1, a field for the target material (ObjectTexture) can be added to describe the material of the detected obstacle:

[0105] Table 1 Target interface description (partial)

[0106]

[0107] Of course, in some possible implementations, the above obstacle detection result may also include the texture of the obstacle, so that the server may classify the candidate objects based on at least one of the material feature and the texture feature.

[0108] In scenarios such as autonomous driving or assisted driving, when the obstacle detection result includes the location and material of the obstacle, the server can also determine the drivable area based on the location and material of the obstacle, and then send the drivable area to the vehicle controller to instruct the vehicle to drive according to the drivable area. Of course, the server can warn the user based on the location and material of the obstacle, reminding the driver whether the obstacle exists on the driving path.

[0109] As can be seen from the above, the embodiment of the present application provides an obstacle detection method, in which a hyperspectral image is obtained by reconstructing a first image encoded based on a three-primary color light pattern, and a hyperspectral feature is extracted from the hyperspectral image. Since the hyperspectral features corresponding to different materials are different, the candidate objects in the hyperspectral image are classified based on the hyperspectral features, and objects with similar colors but different materials can be distinguished. Based on this, obstacle detection with the same or similar color as the environment can be achieved, and the missed detection rate can be reduced. In addition, the method can obtain a hyperspectral image by reconstructing the image using an ordinary camera, without the need for an imaging spectrometer, so the cost is low and it has high feasibility.

[0110] In order to make the technical solution of the present application clearer and easier to understand, the obstacle detection method will be introduced from the perspective of module interaction below.

[0111] See also Figure 5 The flowchart of the obstacle detection method shown in FIG. 1 includes:

[0112] 1. The Camera module obtains RGB images;

[0113] The camera in the Camera module can be a common camera, which can reduce the hardware cost;

[0114] 2. The Camera module sends RGB images to the hyperspectral module;

[0115] 3. The Camera module extracts spatial features from RGB images;

[0116] The spatial feature may specifically be Freespace information in an RGB image;

[0117] 4. The hyperspectral module reconstructs the RGB image to obtain a hyperspectral image and extracts hyperspectral features from the hyperspectral image;

[0118] 5. The Camera module sends spatial features to the Fusion module;

[0119] 6. The hyperspectral module and fusion module send hyperspectral features;

[0120] The execution order of steps 2, 3 and steps 5, 6 may be arbitrary, for example, they may be executed simultaneously or in sequence according to a set order;

[0121] 7. The fusion module fuses the spatial features and the hyperspectral features to obtain fusion features;

[0122] In one example, the fusion module can fuse two images, namely, the RGB image and the target Bounding Box on the hyperspectral image. The input is the target Bounding Box on the RGB image and the target Bounding Box on the hyperspectral module. The target Bounding Boxes are fused in combination with the properties of the positions and velocities of the targets, thereby realizing the fusion of the spatial features in the RGB image and the hyperspectral features in the hyperspectral image.

[0123] The typical fusion algorithm can be performed using the Bayesian data fusion algorithm. For targets that are not detected in the RGB image because of unclear color features, target detection is achieved in the hyperspectral image. In this way, the target detection results of the hyperspectral image are relied upon during fusion, which can achieve comprehensive detection of the target and reduce missed detection of obstacles.

[0124] 8. The fusion module uses the fusion features to classify the candidate objects in the image and outputs the obstacle detection results;

[0125] The obstacle detection result includes the location and material of the obstacle.

[0126] It should be noted that the above-mentioned hyperspectral module is a logical module. During physical deployment, it can be deployed together with the Camera module or deployed separately.

[0127] In some possible implementations, the hyperspectral module can also reconstruct images based on the data dictionary in the configuration module, and then implement obstacle detection based on the reconstructed hyperspectral images. Figure 6 The flowchart of the obstacle detection method shown in FIG. 1 includes:

[0128] 0. The hyperspectral module obtains the data dictionary used for hyperspectral module reconstruction from the configuration module in advance;

[0129] The data dictionary includes the correspondence between spatial features and spectral features. Therefore, the RGB image can be converted into a hyperspectral image based on the data dictionary and applied to subsequent obstacle detection.

[0130] 1. Get RGB images through the Camera module;

[0131] 2. The Camera module sends RGB images to the hyperspectral module;

[0132] 3. The hyperspectral module reconstructs the RGB image based on the data dictionary to obtain the hyperspectral image;

[0133] 4. The Camera module extracts spatial features from RGB images;

[0134] 5. The hyperspectral module extracts hyperspectral features from hyperspectral images;

[0135] 6. The Camera module sends spatial features to the Fusion module;

[0136] 7. The hyperspectral module sends hyperspectral features to the fusion module;

[0137] 8. The fusion module uses a fusion algorithm to fuse spatial features and hyperspectral features;

[0138] 9. The fusion module classifies the candidate objects based on the fusion features and obtains the obstacle detection results.

[0139] The specific implementation of the relevant steps in this embodiment can be referred to the relevant content description above, which will not be repeated here. It should be noted that the execution order between steps 0 to 7 in this embodiment can be set according to actual needs. For example, steps 0 and 1 can be executed in parallel, and steps 6 and 7 can also be executed in parallel. Of course, the above steps can also be executed in sequence according to the set order.

[0140] The above is a specific implementation method of the obstacle detection method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device, which is introduced below from the perspective of functional modularization.

[0141] See also Figure 7 The schematic diagram of the structure of the obstacle detection device shown in FIG. 700 includes:

[0142] An acquisition module 710 is used to acquire a first image, where the first image is an image encoded based on a three-primary color light mode;

[0143] A reconstruction module 720, configured to reconstruct the first image to obtain a second image, where the second image is a hyperspectral image;

[0144] The detection module 730 is used to extract hyperspectral features from the hyperspectral image, classify candidate objects in the hyperspectral image according to the hyperspectral features, and obtain obstacle detection results.

[0145] Optionally, the reconstruction module 720 is specifically used for:

[0146] extracting spatial features of the first image;

[0147] According to the spatial features of the first image, image reconstruction is performed using the correspondence between the spatial features and the spectral features to obtain a second image.

[0148] Optionally, the acquisition module 710 is further used for:

[0149] Obtaining a data dictionary from a configuration file, wherein the data dictionary includes a correspondence between spatial features and spectral features; or,

[0150] Sample data is obtained, and machine learning is performed using the sample data to obtain a correspondence between spatial features and spectral features.

[0151] Optionally, the device 700 further includes:

[0152] A fusion module, used for fusing the hyperspectral feature and the spatial feature of the first image to obtain a fusion feature;

[0153] The detection module 730 is specifically used for:

[0154] The candidate objects in the hyperspectral image are classified according to the fused features.

[0155] Optionally, the fusion module is specifically used for:

[0156] The hyperspectral features and the spatial features of the first image are fused by a Bayesian data fusion algorithm.

[0157] Optionally, the first image includes an RGB image, an RCCC image, an RCCB image or an RGGB image.

[0158] Optionally, the obstacle detection result includes the location and material of the obstacle;

[0159] The device also includes:

[0160] A determination module, used to determine a drivable area according to the location and material of the obstacle;

[0161] The sending module is used to send the drivable area to the vehicle controller to instruct the vehicle to travel according to the drivable area.

[0162] The embodiment of the present application also provides a device for implementing obstacle detection, which can be a server. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 822 (for example, one or more processors) and memory 832, and one or more storage media 830 (for example, one or more mass storage devices) storing application programs 842 or data 844. Among them, the memory 832 and the storage medium 830 can be short-term storage or persistent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 822 can be configured to communicate with the storage medium 830 and execute a series of instruction operations in the storage medium 830 on the server 800.

[0163] The server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858, and / or one or more operating systems 841, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0164] The steps performed by the server in the above embodiment can be based on the Figure 8 The server structure shown.

[0165] The CPU 822 is used to execute the following steps:

[0166] Acquire a first image, where the first image is an image encoded based on a three-primary color light mode;

[0167] Reconstructing the first image to obtain a second image, where the second image is a hyperspectral image;

[0168] A hyperspectral feature is extracted from the hyperspectral image, and candidate objects in the hyperspectral image are classified according to the hyperspectral feature to obtain an obstacle detection result.

[0169] Optionally, CPU822 is also used to execute the steps of any implementation method of the obstacle detection method provided in the embodiments of the present application.

[0170] It can be understood that the above-mentioned server cooperates with the driving assistance system or the automatic driving system in the vehicle to realize obstacle detection. In some possible implementations, the above-mentioned obstacle detection method can also be independently implemented by the driving assistance system or the automatic driving system. The driving assistance system is used as an example below.

[0171] The present application also provides a driving assistance system, including a processor and a memory:

[0172] The memory is used to store computer programs;

[0173] The processor is configured to perform the following steps according to the instructions in the computer program:

[0174] Acquire a first image, where the first image is an image encoded based on a three-primary color light mode;

[0175] Reconstructing the first image to obtain a second image, where the second image is a hyperspectral image;

[0176] A hyperspectral feature is extracted from the hyperspectral image, and candidate objects in the hyperspectral image are classified according to the hyperspectral feature to obtain an obstacle detection result.

[0177] Optionally, the processor is also used to execute the steps of any implementation method of the obstacle detection method provided in the embodiments of the present application.

[0178] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the obstacle detection method described in the present application.

[0179] The embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on a computer, the computer executes the obstacle detection method described in the above aspects.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0184] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An obstacle detection method, characterized in that: Applied to a driving assistance system, the method comprises: Acquire a first image captured by the vehicle surroundings, where the first image is an image encoded based on a three-primary color light pattern; Acquire a hyperspectral image corresponding to the first image; determining a fusion characteristic based on the first image and the hyperspectral image; Obstacle detection results in the first image are obtained according to the fused feature pairs.

2. The method according to claim 1, characterized in that The acquiring of a hyperspectral image corresponding to the first image comprises: reconstructing the first image to obtain a second image, where the second image is the hyperspectral image; or A third image corresponding to the first image is acquired based on a hyperspectral sensor, where the third image is the hyperspectral image.

3. The method according to claim 2, characterized in that The reconstructing the first image to obtain the second image comprises: extracting spatial features of the first image; According to the spatial features of the first image, image reconstruction is performed using the correspondence between the spatial features and the spectral features to obtain a second image.

4. The method according to claim 3, characterized in that The method further comprises: Acquire a data dictionary from a configuration file, wherein the data dictionary includes a correspondence between the spatial features and the spectral features; or, Sample data is obtained, and machine learning is performed using the sample data to obtain the corresponding relationship between the spatial features and the spectral features.

5. The method according to claim 2, characterized in that: The reconstructing the first image to obtain the second image comprises: The second image is obtained based on the first image and a neural network model, where the neural network model is trained based on images encoded with historical three-primary-color light patterns and historical hyperspectral images.

6. The method according to any one of claims 1 to 5, characterized in that: The determining of the fusion characteristic based on the first image and the hyperspectral image comprises: extracting spatial features of the first image; determining a hyperspectral feature of the hyperspectral image; The fusion feature is obtained based on the hyperspectral feature and the spatial feature of the first image.

7. The method according to any one of claims 1 to 6, characterized in that: Obtaining the obstacle detection result in the first image according to the fusion feature pair includes: The candidate objects in the hyperspectral image are classified according to the fusion features to obtain an obstacle detection result in the first image.

8. The method according to claim 6, characterized in that The obtaining of the fusion feature based on the hyperspectral feature and the spatial feature of the first image comprises: The hyperspectral features and the spatial features of the first image are fused by a Bayesian data fusion algorithm.

9. The method according to any one of claims 1 to 8, characterized in that: The obstacle detection result is used to indicate whether there are obstacles in the surrounding environment of the vehicle.

10. The method according to any one of claims 1 to 9, characterized in that: The first image includes an RGB image, an RCCC image, an RCCB image or an RGGB image.

11. The method according to any one of claims 1 to 10, characterized in that: The obstacle detection result includes the location and material of the obstacle; The method further comprises: Determine a drivable area according to the location and material of the obstacle; The drivable area is sent to a vehicle controller to instruct the vehicle to travel according to the drivable area.

12. A driving assistance system, characterized in that: Including processor and memory: The memory is used to store computer programs; The processor is configured to execute the obstacle detection method according to any one of claims 1 to 11 according to the instructions in the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the obstacle detection method according to any one of claims 1 to 11.

14. A computer program product, characterized in that When the computer program product is executed on a computer, the computer executes the obstacle detection method according to any one of claims 1 to 11.

15. A vehicle, characterized in that: The vehicle includes the driving assistance system as claimed in claim 12.

16. The vehicle according to claim 15, characterized in that The vehicle also includes a controller, which is used to control the vehicle driving according to the obstacle detection result output by the driving assistance system.

Images