Unlearned Area Estimation Device and Integrated Program

Abstract

This invention provides an unlearned area estimation device and integrated program that improve the accuracy of detecting obstacles or fallen objects on roads. [Solution] The unlearned region estimation device includes: a first region estimation unit that estimates the object region of a falling object based on an input signal; a second region estimation unit that estimates a region different from the features of the first region estimation unit based on the same or different signal as the input signal; and an estimation region integration unit that integrates the regions estimated by the first region estimation unit and the second region estimation unit based on the captured scene.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence, and in particular, relates to an unseen region estimation device and an integrated program.

Background Art

[0002] In Advanced Driving Assistance Systems (ADAS) and automated driving, when there are falling objects or obstacles on the road, it is necessary to recognize the falling objects or obstacles in order to travel while avoiding them.

[0003] However, the number of collectable samples of falling objects is small and the samples are highly diverse, so it is difficult to learn falling objects in advance.

[0004] At present, the detection method for falling objects or obstacles on the road is to use a neural network to perform region segmentation on collected images for detection. However, for learned object categories, the neural network will not recognize them as falling objects or obstacles. Therefore, with current methods, it becomes difficult to detect falling objects or dropped objects of the learned object categories, leading to missed detections and reduced detection accuracy.

Summary of the Invention

[0005] The present application provides an unseen region estimation device and an integrated program that can improve the detection accuracy of obstacles or falling objects on roads.

[0006] The technical means of the present application are implemented as follows. In a first aspect, the present application provides an unseen region estimation device, the device comprising a first region estimation unit configured to estimate an object region of a falling object based on an input signal, a second region estimation unit configured to estimate a region different from the feature of the first region estimation unit, based on a signal that is the same as or different from the input signal, The system includes an estimated region integration unit configured to integrate the regions estimated by the first region estimation unit and the second region estimation unit based on the shooting scene.

[0007] Preferably, the unlearned region estimation device includes a region estimation unit, and the estimated region integration unit includes an estimated region integration unit. The region estimation unit is configured to perform region estimation having two or more different features, and the estimated region integration unit is configured to integrate multiple estimated regions.

[0008] Preferably, the estimation region integration unit further includes a shooting scene determination unit, The shooting scene determination unit is configured to determine the scene from which to acquire the input signal, and the estimated region integration unit is configured to integrate the estimated region based on the output from the shooting scene determination unit.

[0009] Preferably, the first region estimation unit is configured to perform image segmentation based on the input signal to estimate the object region of the falling object. The second region estimation unit is configured to perform depth estimation based on the same or different input signal and to estimate a region different from the characteristics of the first region estimation unit.

[0010] Preferably, the second region estimation unit is configured to perform depth estimation based on the same or different input signal to determine a depth value corresponding to a pixel, determine a target object based on the depth value corresponding to the pixel, predict normal vector information corresponding to the pixel, and use the normal vector information to determine a region corresponding to the target object.

[0011] Preferably, the apparatus includes a mixed expert model, The mixed expert model is configured to integrate estimated regions output by multiple region estimation units based on the shooting scene using the estimation region integration unit, and the multiple region estimation units include the first region estimation unit and the second region estimation unit.

[0012] Preferably, the shooting scene determination unit is configured to determine the scene of the input signal based on a scene understanding algorithm.

[0013] Preferably, the unlearned region estimation device includes a plurality of region estimation units, Each of the aforementioned region estimation units is configured to perform region estimation having two or more different features based on the signal it has input.

[0014] Preferably, the input signal includes at least one signal data collected by at least one sensor, including at least one of an image sensor and a laser radar sensor. In a second embodiment, the present application provides an unlearned region estimation method, the method comprising: first region estimation unit estimating the object region of a falling object based on an input signal; The second region estimation unit estimates a region different from the characteristics of the first region estimation unit based on a signal that is the same as or different from the input signal. The system includes integrating the regions estimated by the first region estimation unit and the second region estimation unit based on the shooting scene using an estimated region integration unit.

[0015] Preferably, the method is The region estimation unit performs region estimation with two or more different features, The system further includes integrating multiple estimation regions using the estimation region integration unit in the estimation region integration unit.

[0016] Preferably, the estimation region integration unit further includes a shooting scene determination unit, and integrating a plurality of estimation regions determines the scene from which the input signal is acquired. The estimation region integration unit includes integrating the estimated region based on the output from the shooting scene determination unit.

[0017] Preferably, inferring the object region of a falling object based on an input signal includes performing image segmentation based on the input signal to infer the object region of the falling object. Inferring a region different from the characteristics of the first region estimation unit based on a signal that is the same as or different from the input signal includes performing depth estimation based on a signal that is the same as or different from the input signal to infer a region different from the characteristics of the first region estimation unit.

[0018] Preferably, depth estimation is performed based on the same or different signals as the input signal to estimate a region different from the characteristics of the first region estimation unit. Depth estimation is performed based on the same or different input signal to determine the depth value corresponding to the pixel. This includes determining a target object based on the depth value corresponding to a pixel, predicting normal vector information corresponding to the pixel, and determining a region corresponding to the target object using the normal vector information.

[0019] Preferably, determining the scene from which to acquire the input signal includes determining the scene of the input signal based on a scene understanding algorithm.

[0020] In a third embodiment, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores executable instructions. When the processor executes an executable instruction stored in the memory, it implements the unlearned area estimation method according to the embodiment of the present invention.

[0021] In a fourth embodiment, the present application provides a computer-readable storage medium that stores executable instructions, when executed by a processor, that implement an unlearned area estimation method according to an embodiment of the present application.

[0022] In a fifth aspect, the embodiments of the present application provide an integrated program that includes a computer program or instructions, which when executed by a processor, implements the unseen region estimation method according to the embodiments of the present application.

[0023] The present application provides an unseen region estimation apparatus and an integrated program, wherein a first region inference unit infers an object region of a falling object based on an input signal, a second region inference unit infers a region with features different from those of the first region inference unit based on a signal that is the same as or different from the input signal, and an inferred region integration unit can integrate the regions inferred by the first region inference unit and the second region inference unit based on an imaging scene. In the embodiments of the present application, the first region inference unit and the second region inference unit respectively infer the object region of the falling object based on the input signal and the signal that is the same as or different from the input signal to obtain regions with different features, and the inferred region integration unit integrates the region inferred by the first region inference unit and the region inferred by the second region inference unit based on the imaging scene, and adaptively combines the regions of different features inferred by different inference units based on the imaging scene to estimate the object region. This enables complementary use of different feature inference methods to detect obstacles or falling objects on the road, reduce detection omissions, and improve the detection accuracy of obstacles or falling objects. [Brief Description of the Drawings]

[0024] [Figure 1] This is an example of a forward road surface image collected by a vehicle-mounted camera. [Figure 2] This is a classification prediction diagram of a current neural network for forward road surface images. [Figure 3] This is an optional flowchart of the unseen region estimation method according to the embodiments of the present application. [Figure 4] This is an optional schematic diagram of a depth map output by the second region inference unit according to the embodiments of the present application. [Figure 5] This is a schematic diagram of an optional network processing flow of the unseen region estimation method according to the embodiments of the present application. [Figure 6]This is a schematic diagram of the selectable network processing flow of the unlearned area estimation method according to an embodiment of the present application. [Figure 7] This is a schematic diagram of the selectable network processing flow of the unlearned area estimation method according to an embodiment of the present application. [Figure 8] This is a schematic diagram of the selectable network processing flow of the unlearned area estimation method according to an embodiment of the present application. [Figure 9] This is a schematic diagram of the selectable network processing flow of the unlearned area estimation method according to an embodiment of the present application. [Figure 10] This is a schematic diagram of selectable configurations for an unlearned area estimation device according to an embodiment of the present application. [Figure 11] This is a schematic diagram of selectable configurations of an electronic device according to an embodiment of the present application. [Modes for carrying out the invention]

[0025] To further clarify the purpose, technical means and advantages of this application, the application will be described in more detail below with reference to the drawings, and the embodiments described therein should not be considered limiting to this application, and all other embodiments obtained by a person skilled in the art without creative work are all within the scope of protection of this application.

[0026] In the following description, "some embodiments" refers to a subset of all possible embodiments, but it should be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and can be combined with each other if they do not contradict each other.

[0027] In the following description, the related terms “First / Second / Third” are merely used to distinguish similar subjects and do not represent a specific order of subjects. It should be understood that “First / Second / Third” may be interchangeable with any particular order or sequence, as permitted, so that the embodiments of the present application described herein may be carried out in an order other than that shown or described.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used herein are for illustrative purposes only and are not intended to limit the application.

[0029] Currently, the method for detecting fallen objects or obstacles on roads involves using a neural network to segment the collected image and perform detection. However, the neural network does not recognize objects in the learned object categories as fallen objects or obstacles. For example, in Figure 1, the obstacles on the road surface in a forward road image collected by a vehicle's camera are bicycles 11 and 12. Because the neural network has learned the features of bicycles, it recognizes bicycle 11, which has clear visual features, and in the output classification prediction diagram shown in Figure 2, it classifies the pixels of bicycle 11 as bicycles, forming the image region 21 classified as bicycles in Figure 2. Because the appearance of bicycle 12 is significantly different from the appearance of bicycles learned by the neural network, it is considered a fallen object or obstacle, forming the image region 22 classified as an obstacle in Figure 2. It is clear that the current method has difficulty detecting fallen objects or objects in the learned object categories, which can lead to detection failures, a decrease in detection accuracy, and potentially a decrease in the safety of autonomous or assisted driving.

[0030] The embodiment of the present invention provides an unlearned area estimation device and integrated program that can improve the detection accuracy of fallen objects or obstacles on roads. The unlearned area estimation method of the embodiment of the present invention includes the following S101 to S103, as shown in Figure 3.

[0031] In S101, the first region estimation unit estimates the object region of the falling object based on the input signal.

[0032] In S102, the second region estimation unit estimates a region different from the characteristics of the first region estimation unit, based on a signal that is the same as or different from the input signal.

[0033] In the embodiments of this application, the input signal may include an image collected by an image sensor or point cloud data scanned by a laser radar. The first or second region estimation unit may include one of the following neural networks: a neural network for image segmentation tasks, a neural network for depth estimation, or at least one neural network for target detection based on signals from at least one sensor. The first and second region estimation units are neural networks that perform object region estimation of fallen objects on a road using different feature estimation methods. In other words, the first and second region estimation units are neural networks with different feature processing processes.

[0034] In some embodiments, the first region inference unit may be a neural network for the image segmentation task. The first region inference unit can perform the image segmentation task by extracting features from an image, distinguishing at least one image region corresponding to at least one type of object in the image based on the extracted features, and outputting a category corresponding to each pixel point in the image. Exemplarily, the first region inference unit may perform pixel-level semantic segmentation on an image and output a semantic segmentation map, the semantic segmentation map including semantic category labels corresponding to each pixel point in the input image. Exemplarily, the first region inference unit may include a neural network with a Mask2Former structure.

[0035] The first region prediction unit of the image segmentation task can recognize and divide image regions corresponding to learned objects from the input image. If the image contains an object not yet learned by the first region prediction unit, the first region prediction unit can divide the image region corresponding to the unlearned object and label the category of pixels contained in that image region as a category representing unlearned objects (or a category representing anomalies). In other words, by detecting object regions corresponding to unlearned objects in an image using the first region prediction unit, it is possible to detect unlearned obstacles or fallen objects that appear on the road.

[0036] In the embodiments of the present application, the signals processed by the second region estimation unit may include the same or different signals as the input signals of the first region estimation unit. For example, the same image can be processed using the first and second region estimation units, or images collected on a road at different viewing angles can be processed using the first and second region estimation units, respectively.

[0037] In some embodiments, the second domain estimation unit is a neural network that performs a depth estimation task. The second domain estimation unit performs depth estimation on an input image and outputs depth information corresponding to each pixel in the image. Based on the depth information corresponding to each pixel, it can recognize three-dimensional objects in the image and determine three-dimensional objects present on the road, thereby enabling the detection of obstacles or fallen objects on the road. Exemplarily, the depth information corresponding to a pixel may include the relative distance between the pixel and the image source.

[0038] For example, the second region estimation unit may include a depth estimation network based on monocular or binocular vision. The second region estimation unit can output a depth map based on the depth information of each pixel, and if the second region estimation unit performs depth estimation on Figure 1, the output depth map may be as shown in Figure 4.

[0039] By detecting or recognizing three-dimensional objects 41 and 42 on the vehicle's travel route based on depth information corresponding to each pixel in Figure 4, it is possible to detect obstacles or fallen objects on the travel route. In the embodiment of this application, the second region estimation unit outputs a region that has different characteristics from the region output by the first region estimation unit. For example, the object region output by the first region estimation unit is a region representing the semantic characteristics of the fallen object. The region output by the second region estimation unit is a region representing the depth characteristics of the fallen object. Alternatively, the first and second region estimation units can perform target detection on two-dimensional images and three-dimensional point cloud data collected by different sensors to represent the object regions of fallen objects with different characteristics, and output different types of target detection regions (bounding boxes).

[0040] In S103, the estimated region integration unit integrates the regions estimated by the first region estimation unit and the second region estimation unit based on the shooting scene.

[0041] In the embodiments of the present application, the estimated region integration unit integrates the regions estimated by the first region estimation unit and the second region estimation unit based on the shooting scene. The shooting scene includes scene elements for signal acquisition from a falling object. The scene elements may include object feature elements of the falling object to be acquired, or acquisition environment elements where the signal acquisition takes place. In some embodiments, the shooting scene may include the appearance of the falling object (e.g., shape and / or size of the falling object), the distance between the falling object and the electronic device, or the shooting environment in which the falling object is currently located (e.g., weather, light intensity, etc.). Specifically, these are selected based on actual circumstances and are not limited to the embodiments of the present application.

[0042] In some embodiments, the inference region integration unit may include a shooting scene determination unit, and exemplary the shooting scene determination unit may include a neural network for determining the shooting scene, and the input signal to the inference region integration unit may be the same signal as that of the first region inference unit and / or the second region inference unit, and the shooting scene determination unit in the inference region integration unit determines the shooting scene using a scene understanding algorithm based on the signal input to the first region inference unit and / or the signal input to the second region inference unit. Alternatively, the inference region integration unit may be connected to an independent shooting scene determination unit. Exemplarily the input signal to the inference region integration unit may be the output signal of the shooting scene determination unit, and the input signal to the shooting scene determination unit may be the same signal as that of the first region inference unit and / or the second region inference unit, thereby determining the shooting scene using a scene understanding algorithm based on the signal input to the first region inference unit and / or the signal input to the second region inference unit and outputting it to the inference region integration unit.

[0043] In current related technologies, the problem of integrating two different network prediction results into one is generally addressed by using a simple threshold. However, since the threshold belongs to the hyperparameters of the neural network, determining an appropriate threshold for integration is difficult. Furthermore, because the input environments differ and the thresholds applied also differ, the simple threshold integration methods of current related technologies are generally not suitable for practical applications.

[0044] In the embodiments of this application, the inference region integration unit may be implemented in the form of a neural network. The inference region integration unit may be trained together with an untrained first region inference unit and an untrained second region inference unit, or it may be trained based on a pre-trained first region inference unit and a second region inference unit, specifically selected based on the actual situation and not limited to the embodiments of this application. During network training, the inference region integration unit learns the optimal parameters for integrating the regions output by the first region inference unit and the regions output by the second region inference unit in various shooting scenes. In this way, after completing training, the inference region integration unit can be used to automatically and optimally integrate the regions inferred by the first region inference unit and the regions inferred by the second region inference unit based on the shooting scene, thereby enabling accurate detection of regions where falling objects or obstacles exist. Furthermore, the system's robustness can be improved by adaptively adjusting the optimal parameters for region integration in different shooting scenes.

[0045] It should be understood that the embodiment of this application uses a first-region estimation unit and a second-region estimation unit to estimate the object region of a falling object based on an input signal and a signal that is the same as or different from the input signal, respectively, thereby acquiring regions with different features, and the estimation region integration unit integrates the region estimated by the first-region estimation unit and the region estimated by the second-region estimation unit based on the shooting scene, thereby adaptively combining regions with different features estimated by different estimation units based on the shooting scene to estimate the object region. In this way, it is possible to detect obstacles or falling objects on a road by complementaryly utilizing different feature estimation methods, reduce detection omissions, and improve the detection accuracy of obstacles or falling objects.

[0046] In some embodiments, the neural network model including a first domain inference unit, a second domain inference unit, and an inference domain integration unit may be a Mixture of Expert (MoE). The first and second domain inference units correspond to expert models in the MoE, and the inference domain integration unit corresponds to a gating network or gate model in the MoE. The gating network or gate model balances and integrates the outputs of the expert models, determines the weights of each expert model for the final prediction, and integrates the results. For example, the first domain prediction unit may include a semantic segmentation neural network, and the second domain prediction unit may include a depth estimation neural network. In the case of a situation of a fallen object on a road, such as object 50 in Figure 5, if object 50 differs significantly from the appearance of the object learned by the semantic segmentation neural network 52, the prediction domain integration unit 51 may assign higher weights to the region output by the semantic segmentation neural network 52 between the semantic segmentation neural network 52 and the depth estimation neural network 53, integrate the regions output by the semantic segmentation neural network 52 and the depth estimation neural network 53 based on the assigned weights, and finally output the determined object region 54 of the fallen object. In the case of objects falling on the road, such as objects 60-1 and 60-2 in Figure 6, if objects 60-1 and 60-2 differ slightly from the appearance of the objects learned by the semantic segmentation neural network 62, the inferred region integration unit 61 may assign higher weights to the regions output by the depth estimation neural network 63 between the semantic segmentation neural network 62 and the depth estimation neural network 63, integrate the regions output by the semantic segmentation neural network 62 and the depth estimation neural network 63 based on the assigned weights, and finally output the determined object regions 64-1 and 64-2 of the fallen objects.

[0047] In some embodiments, the filming scene may include the shape and / or size of the falling object.

[0048] In some embodiments, the estimated region integration unit may adaptively select high-weight feature information output by the region estimation unit on a pixel-by-pixel basis from the region output by the first region estimation unit and the region output by the second region estimation unit, based on the shape and / or size of the falling object, integrate using a weighted sum, and output the object region of the falling object.

[0049] For example, as shown in Figure 7, the input signals to the first region inference unit and the signals to the second region inference unit may be the same color (RGB) image, and the first region inference unit may include a semantic segmentation network 71, which determines a pixel label corresponding to each pixel in the RGB image by semantic segmentation, and the pixel label indicates whether or not the pixel belongs to a falling object (an unlearned object). In some embodiments, the pixel label may include an anomaly score for the pixel. For example, the anomaly score corresponding to a pixel represents the probability that the pixel is a falling object. The greater the difference between the appearance of the pixel and the object category learned by the semantic segmentation network, the higher the probability that the pixel is a falling object.

[0050] Here, the pixel labels corresponding to each pixel in an RGB image may be considered as pixel label images output by a semantic segmentation network.

[0051] As shown in Figure 7, the second region estimation unit may include an object detection network 72 based on monocular depth estimation and normal vectors. The object detection network 72 obtains depth values ​​corresponding to each pixel in the RGB image using a monocular depth estimation (e.g., MonoDepth depth estimation) algorithm, and determines depth information corresponding to each pixel based on the depth values. In some embodiments, the object detection network 72 can predict normal vector information corresponding to each pixel in the image, and exemplary, the normal vector information may include the surface normal vector of the object on which the pixel is located. Using the depth values ​​corresponding to each pixel, a target object can be tentatively determined, and using the normal vector information, the region corresponding to the target object can be further extracted. In other words, by combining the normal vector information and depth values ​​to determine the depth information corresponding to each pixel, the depth information image output by the object detection network 72 is determined, and the depth information image includes the estimated region obtained by the depth estimation of the second region estimation unit. In this way, the depth values ​​are post-processed or optimized using the surface normal vectors. For example, by optimizing the depth value using normal vector information through smoothness constraints and surface continuity constraints, it is possible to ensure consistency between the surface normal vector and the depth value.

[0052] As shown in Figure 7, the inference region integration unit may include a neural network having an encoder-decoder structure (e.g., UNet), such as a gate model 73. Exemplaryly, the gate model 73 can acquire weights w1 corresponding to the semantic segmentation network 71 and weights w2 corresponding to the object detection network 72 through supervised learning during training. In this way, the gate model 73 can integrate the pixel labels and depth information on a pixel-by-pixel basis, weighting them with w1 and w2 respectively, based on the pixel label image output by the semantic segmentation network 71 and the depth information image output by the object detection network 72, and output the integrated result, which includes the object region of the finally determined falling object.

[0053] In the embodiment of this invention, since objects not included in the training data are recognized as falling objects, it should be understood that a large amount of training data for different types of falling objects is not required at all. Furthermore, by integrating object detection information based on depth estimation, the problem of false detection of objects similar to trained objects in untrained object detection models is avoided, and the accuracy of obstacle or falling object detection is improved.

[0054] In some embodiments, the first and second region estimation units may be region estimation units among a plurality of region estimation units, and the plurality of region estimation units may further include other region estimation units. The plurality of region estimation units may output estimated regions of multiple different features. The estimated region integration unit may integrate the regions of multiple different features output by the plurality of region estimation units. Exemplarily, the estimated region integration unit integrates the plurality of estimated regions, performs region estimation having two or more different features, and determines the final object region of the falling object.

[0055] In some embodiments, the shooting scene may include the shooting environment. Exemplarily, the shooting environment may include the weather and / or lighting conditions at the time of shooting, such as a sunny day, night, daytime, or rainy day, and will be specifically selected based on the actual circumstances, and is not limited to the embodiments of this application.

[0056] In some embodiments, the prediction region integration unit may adaptively select the output of the computationally efficient region prediction unit from among multiple region prediction units based on the captured scene to achieve high-performance prediction. Each of the multiple region prediction units may be a computationally efficient lightweight model (e.g., MobileNet), and each region prediction unit excels at detecting falling objects in at least one type of shooting environment. The prediction region integration unit may be a lightweight model (e.g., a gate model) having the same structure, and may output weights corresponding to each region prediction unit in different shooting environments by automatically learning during training which shooting environments each of the multiple region prediction units excels at processing.

[0057] For example, the corresponding weight in the shooting environment may represent the importance (which may be a probability value) of the output result of the corresponding estimated region integration unit in that shooting environment. For example, as shown in Figure 8, the gate model can output weights corresponding to each region estimation unit for images in each shooting environment, based on various shooting environments (daytime on a sunny day, twilight on a rainy day, night on a sunny day, night on a rainy day, daytime on a foggy day, etc.) and the shooting environment in which each learned region estimation unit (region estimation unit 81, region estimation unit 82, region estimation unit 83) excels at processing. For an image in a daytime environment on a sunny day, weight w1 corresponding to region estimation unit 81, weight w2 corresponding to region estimation unit 82, and weight w3 corresponding to region estimation unit 83 are output. In some embodiments, not using the estimated region output by the corresponding region estimation unit can be represented by setting the weight to 0. Based on the weights corresponding to each region estimation unit, the estimated regions output by the region estimation unit are weighted, and multiple weighted estimated regions corresponding to multiple region estimation units are integrated. For example, the estimated region output by region estimation unit 81 is weighted by w1 to obtain weighted estimated region 1 corresponding to region estimation unit 81, the estimated region output by region estimation unit 82 is weighted by w2 to obtain weighted estimated region 2 corresponding to region estimation unit 82, the estimated region output by region estimation unit 83 is weighted by w3 to obtain weighted estimated region 3 corresponding to region estimation unit 83, weighted estimated region 1, weighted estimated region 2, and weighted estimated region 3 are integrated and combined, and based on the combined result, the object region of the falling object in the shooting scene is determined, and furthermore, the detection of falling objects in various different shooting environments can be processed.

[0058] Furthermore, even when the shooting scene includes the shooting environment, multiple region estimation units and estimation region integration units may be implemented using the MoE model. The multiple region estimation units correspond to expert models in the MoE model, and the estimation region integration unit corresponds to a gate model or gating network in the MoE model.

[0059] Compared to current related technologies, it should be understood that while object detection in multi-shooting environments is typically achieved using one large model, the present invention's embodiment improves the overall computational efficiency by using multiple smaller models, thereby improving the detection efficiency of falling objects. Furthermore, since the optimal model can be selected and output based on the input shooting environment, there is no need to set new thresholds. In addition, it is possible to prevent accuracy degradation due to changes in the shooting environment and achieve more robust detection.

[0060] In some embodiments, the shooting scene may include the distance between the falling object and the electronic device. Multiple region estimation units may include multiple region estimation units corresponding to various sensor types. Exemplarily, the sensor type may include image sensors such as cameras, fisheye cameras, and laser radars. Correspondingly, the input signal may include sensor signals corresponding to various sensor types such as RGB images from cameras, point clouds from LiDAR, images from fisheye cameras, and 4Dlarar. For each sensor signal of each sensor type, target detection of the falling object is performed by the corresponding region estimation unit among the multiple region estimation units, and the target detection region corresponding to each region estimation unit is output as the estimated region corresponding to each region estimation unit. It should be understood that target detection is performed for different types of input signals, and the format of the target detection region output by different region estimation units may differ. For example, the target detection region corresponding to an RGB image from a camera may be a 2D target detection frame and a 3D target detection frame, and the target detection region corresponding to radar point cloud data may be a 3D target detection frame. In the embodiment of the present invention, the estimation region integration unit can integrate the target detection regions output by different region estimation units to determine the final object region of the falling object.

[0061] In some embodiments, the estimated region integration unit can determine the weights corresponding to each region estimation unit by automatically determining the importance of different types of sensor signals based on the distance between the falling object and the electronic device, and then weight and integrate multiple target detection regions output by multiple region estimation units based on the weights corresponding to each region estimation unit to determine the final object region of the falling object.

[0062] Exemplary, as shown in Figure 9, the multiple region estimation units may include a first network model 91, a second network model 92, and a third network model 93. The estimation region integration unit may include a gate model 94. The first network model 91 processes RGB images collected by a monocular camera, the second network model 92 processes point cloud data collected by a laser radar, and the third network model 93 processes fisheye images collected by a fisheye camera. In other words, each region estimation unit is a neural network model corresponding to its input signal, and the estimation region integration unit is a model capable of processing multimodal inputs. The first network model 91 performs target detection of falling objects on RGB images collected by a monocular camera and outputs a 2D target detection frame corresponding to the RGB image. The second network model 92 performs target detection of falling objects on point cloud data collected by a laser radar and outputs a 3D target detection frame corresponding to the point cloud data. The third network model 93 performs target detection of falling objects on fisheye images collected by a fisheye camera and outputs a 2D target detection frame corresponding to the fisheye image. The multimodal inputs of gate model 94 are RGB images collected by a monocular camera, point cloud data collected by a laser radar, and fisheye images collected by a fisheye camera. Based on the multimodal inputs, gate model 94 determines the distance between the electronic device and the falling object, and based on the distance, determines and integrates one or more target detection frames from the target detection frames output from multiple models to output the final object region of the falling object.

[0063] In some embodiments, the first network model 91 can simultaneously output 2D target detection frames and 3D target detection frames corresponding to RGB images and participate in the determination of the target detection frame. Specifically, the frame is selected based on the actual situation and is not limited to the embodiments of this application.

[0064] It should be understood that each sensor model output has advantages and disadvantages, and the inference region integration unit can improve the accuracy of obstacle or falling object detection by automatically selecting the appropriate model output based on the distance between the electronic device and the falling object to determine the final object region of the falling object. Furthermore, each region inference unit in multiple region inference units, for example, each sensor model, may be implemented by a computationally efficient lightweight model, and the inference region integration unit, for example, the gate model, may also be implemented by a lightweight model. In this way, multiple region inference units and the inference region integration unit can learn synchronously, and by training all models at once in a supervised training manner, for example, the consumption of computational resources can be reduced and the efficiency of network training can be improved.

[0065] The embodiment of the present application further provides an unlearned region estimation device, and Figure 10 is a schematic configuration diagram of the unlearned region estimation device according to the embodiment of the present application.

[0066] As shown in Figure 10, the unlearned region estimation device 100 includes a first region estimation unit 110, a second region estimation unit 120, and an estimation region integration unit 130. The first region estimation unit 110 is configured to estimate the object region of a falling object based on an input signal, and the second region estimation unit 120 is configured to estimate a region different from the features of the first region estimation unit 110 based on the same or different input signal. The estimated region integration unit 130 is configured to integrate the regions estimated by the first region estimation unit 110 and the second region estimation unit 120 based on the shooting scene.

[0067] In some embodiments, the unlearned region estimation device 100 includes a region estimation unit, and the estimated region integration unit 130 includes an estimated region integration unit. The region estimation unit is configured to perform region estimation having two or more different features as described above, and the estimated region integration unit is configured to integrate multiple estimated regions.

[0068] In some embodiments, the estimation region integration unit 130 further includes a shooting scene determination unit, which is configured to determine the scene from which to acquire the above-mentioned input signal. The estimation region integration unit is configured to integrate the estimation region based on the output from the shooting scene determination unit.

[0069] In some embodiments, the first region estimation unit 110 is configured to perform image segmentation based on the input signal to estimate the object region of the falling object. The second region estimation unit 120 is configured to perform depth estimation based on the same or different input signal and to estimate a region different from the characteristics of the first region estimation unit.

[0070] In some embodiments, the second region estimation unit 120 is configured to perform depth estimation based on the same or different input signal to determine a depth value corresponding to a pixel, determine a target object based on the depth value corresponding to the pixel, predict normal vector information corresponding to the pixel, and use the normal vector information to determine a region corresponding to the target object.

[0071] In some embodiments, the unlearned region estimation device 100 includes a mixed expert model, The mixed expert model is configured to integrate the estimated regions output by multiple region estimation units based on the shooting scene using the estimation region integration unit 130, and the multiple region estimation units include the first region estimation unit 110 and the second region estimation unit 120.

[0072] In some embodiments, the shooting scene determination unit is configured to determine the scene of the input signal described above based on a scene understanding algorithm.

[0073] In some embodiments, the unlearned region estimation device 100 includes a plurality of region estimation units, Each of the aforementioned region estimation units is configured to perform region estimation having two or more different features based on the signal it has input.

[0074] In some embodiments, the input signal includes at least one signal data collected by at least one sensor, which includes at least one of an image sensor and a laser radar sensor.

[0075] The above description of the embodiments of the apparatus is similar to the description of the embodiments of the method described above, and has similar beneficial effects. Technical details not disclosed in the embodiments of the apparatus of this application should be understood by referring to the description of the embodiments of the method of this application.

[0076] Embodiments of the present application further provide electronic devices, and Figure 11 is a schematic diagram of a selectable configuration of an electronic device according to an embodiment of the present application. As shown in Figure 11, the electronic device 3 includes a memory 32 and a processor 33.

[0077] The memory 32 and the processor 33 are connected via a communication bus 34. The memory 32 stores executable instructions, and the processor 33 implements the unlearned area estimation method according to the embodiment of the present invention when executing executable instructions stored in the memory 32.

[0078] The embodiment of the present invention provides a computer-readable storage medium that, when executed by the processor described above, stores executable instructions that cause the processor described above to execute the unlearned area estimation method according to the embodiment of the present invention.

[0079] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM®), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or CD-ROM, and may be a variety of devices containing one or any combination of the above-mentioned memories.

[0080] In some embodiments, executable instructions may be written in the form of a program, software, software module, script, or code in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), may be implemented in any form, may be implemented as a standalone program, or may be implemented as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0081] For example, an executable instruction may, but does not necessarily, correspond to a file in a file system, and may be stored in part of a file that holds other programs or data, such as one or more scripts in a Hyper Text Markup Language (HTML) document, a single file dedicated to the program under discussion, or multiple collaborative files (e.g., files that store one or more modules, subprograms, or code sections). For example, an executable instruction may be implemented to run on a single computer, or on multiple computers located in one place, or on multiple computers distributed in multiple locations and interconnected via a communication network.

[0082] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or integrated programs (computer program products). Accordingly, this application may be provided in the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, this application may be provided in the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk memory and optical memory) containing computer-usable program code.

[0083] This application will be described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products relating to embodiments of this application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a dedicated computer, an embedded processing unit or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device can generate a device that realizes the functions defined in one or more flows in the flowchart and / or one or more blocks in the block diagrams.

[0084] These computer program instructions can also be stored in computer-readable memory that can guide a computer or other programmable data processing device to function in a particular manner, and the instructions stored in said computer-readable memory can generate a product that includes an instruction unit that implements functions defined in one or more flows in a flowchart and / or one or more blocks in a block diagram.

[0085] These computer program instructions can also be loaded into a computer or other programmable data processing device, and when a series of operational steps are performed in the computer or other programmable device to generate processing realized by the computer, the instructions executed on the computer or other programmable device provide steps to realize functions defined in one or more flows in a flowchart and / or one or more blocks in a block diagram.

[0086] The above description is merely a preferred embodiment of the present application and does not limit the scope of protection of the present application.

[0087] Any modifications, equivalent substitutions, and improvements made in the spirit and scope of this application are all included within the scope of protection of this application.

Claims

1. A first region estimation unit configured to estimate the object region of a falling object based on an input signal, A second region estimation unit is configured to estimate a region different from the characteristics of the first region estimation unit based on a signal that is the same as or different from the input signal, An unlearned region estimation device, comprising: an estimated region integration unit configured to integrate the regions estimated by the first region estimation unit and the second region estimation unit based on the shooting scene.

2. The region estimation unit includes, and the estimation region integration unit includes the estimation region integration unit. The apparatus according to claim 1, wherein the region estimation unit is configured to perform region estimation having two or more different features, and the estimated region integration unit is configured to integrate a plurality of estimated regions.

3. The aforementioned estimation region integration unit further includes a shooting scene determination unit, The aforementioned shooting scene determination unit is configured to determine the scene from which to acquire the input signal. The apparatus according to claim 2, wherein the estimation region integration unit is configured to integrate the estimation region based on the output from the shooting scene determination unit.

4. The first region estimation unit is configured to estimate the object region of a falling object by performing image segmentation based on the input signal. The apparatus according to claim 1, wherein the second region estimation unit is configured to perform depth estimation based on the same or different signal as the input signal and to estimate a region different from the characteristics of the first region estimation unit.

5. The apparatus according to claim 4, wherein the second region estimation unit is configured to perform depth estimation based on a signal that is the same as or different from the input signal, determine a depth value corresponding to a pixel, determine a target object based on the depth value corresponding to the pixel, predict normal vector information corresponding to the pixel, and determine a region corresponding to the target object using the normal vector information.

6. Includes mixed expert models, The apparatus according to claim 1, wherein the mixed expert model is configured to integrate estimated regions output by a plurality of region estimation units based on the shooting scene using the estimation region integration unit, and the plurality of region estimation units include the first region estimation unit and the second region estimation unit.

7. The apparatus according to claim 3, wherein the shooting scene determination unit is configured to determine the scene of the input signal based on a scene understanding algorithm.

8. The apparatus according to claim 2, comprising a plurality of region estimation units, each of the plurality of region estimation units configured to perform region estimation having two or more different features based on a signal it has input.

9. The apparatus according to claim 1, wherein the input signal includes at least one type of signal data collected by at least one sensor, which includes at least one of an image sensor and a laser radar sensor.

10. The first region estimation unit estimates the object region of the falling object based on the input signal, The second region estimation unit estimates a region different from the characteristics of the first region estimation unit based on a signal that is the same as or different from the input signal. An unlearned region estimation method, comprising: integrating the regions estimated by the first region estimation unit and the second region estimation unit based on the captured scene using an estimation region integration unit.

11. The region estimation unit performs region estimation with two or more different features, The method according to claim 10, further comprising integrating a plurality of estimation regions using an estimation region integration unit in the estimation region integration unit.

12. The aforementioned estimation region integration unit further includes a shooting scene determination unit, and integrating multiple estimation regions determines the scene from which the input signal is acquired. The method according to claim 11, further comprising: integrating the estimated region based on the output from the shooting scene determination unit using the estimated region integration unit.

13. Inferring the object region of a falling object based on an input signal includes performing image segmentation based on the input signal to infer the object region of the falling object. Inferring a region different from the characteristics of the first region estimation unit based on a signal that is the same as or different from the input signal is: The method according to claim 10, comprising performing depth estimation based on a signal that is the same as or different from the input signal, and inferring a region that is different from the characteristics of the first region estimation unit.

14. Performing depth estimation based on a signal that is the same as or different from the input signal, and inferring a region that is different from the characteristics of the first region estimation unit, is: Depth estimation is performed based on the same or different input signal to determine the depth value corresponding to the pixel. The method according to claim 13, comprising: determining a target object based on a depth value corresponding to a pixel, predicting normal vector information corresponding to the pixel, and determining a region corresponding to the target object using the normal vector information.

15. The method according to claim 12, wherein determining the scene in which the input signal is acquired includes determining the scene of the input signal based on a scene understanding algorithm.

16. It includes memory and a processor, the memory storing executable instructions, An electronic device that, when the processor executes an executable instruction stored in the memory, implements the method according to any one of claims 10 to 15.

17. A computer-readable storage medium storing executable instructions that, when executed by a processor, implement the method described in any one of claims 10 to 15.

18. An integrated program that includes computer executable instructions, wherein when the computer executable instructions are executed by a processor, it implements the method according to any one of claims 10 to 15.

Images