System and method for simultaneous real-time object detection and semantic segmentation

Abstract

A system and method for simultaneously performing object detection and semantic segmentation. The system includes a computing device having a processor and non-volatile memory storing computer-executable code. When executed by the processor, the computer-executable code is configured to: receive an image of a scene; process the image using a neural network backbone to obtain a feature map; process the feature map using an object detection module to obtain an object detection result for the image; and process the feature map using a semantic segmentation module to obtain a semantic segmentation result for the image. The object detection module and the semantic segmentation module are trained using the same loss function, which includes both object detection and semantic segmentation components.

Description

Technical Field

[0001] The present disclosure relates generally to computer vision, and more particularly to a system and method for simultaneous real-time object detection and semantic segmentation. Background Art

[0002] The background description provided herein is intended to generally present the context of the present disclosure. To the extent described in this background section, the work of the inventors and the description of aspects that may not qualify as prior art at the time of filing the application are neither explicitly nor implicitly admitted to be prior art with respect to the present disclosure.

[0003] For autonomous vehicles, it is crucial to keep up to date with their surroundings, including cars, pedestrians, traffic lights, traffic signs, and available space. To achieve this, autonomous driving systems typically need to perform multiple vision-based tasks, such as object detection, semantic segmentation, motion estimation, and 3D reconstruction, with object detection and semantic segmentation being two of the most important. However, given the limited computing power of vehicles, the computational cost of performing each task is quite high.

[0004] Therefore, there is a need in the art to address the above-mentioned deficiencies and inadequacies. Summary of the Invention

[0005] In some aspects, the present disclosure relates to an object detection and semantic segmentation system. In some embodiments, the system includes a computing device. The computing device has a processor and non-volatile memory storing computer-executable code. The computer-executable code, when executed at the processor, is configured to: receive an image of a scene; process the image using a neural network backbone to obtain a feature map; process the feature map using an object detection module to obtain an object detection result for the image; and process the feature map using a semantic segmentation module to obtain a semantic segmentation result for the image. The object detection module and the semantic segmentation module are trained using the same loss function, the loss function including an object detection component and a semantic segmentation component.

[0006] In some embodiments, the training data for the object detection module and the semantic segmentation module includes: a training image, at least one bounding box defined in the training image, a label of the at least one bounding box, and a mask of the training image.

[0007] In some embodiments, the neural network backbone comprises a ResNet18 backbone truncated from its 4th block.

[0008] In some embodiments, the object detection module is a single shot detector (SSD).

[0009] In some embodiments, the object detection module includes, in sequence: five convolutional layers; a detection layer; and a non-maximum suppression (NMS) layer. In some embodiments, for a 512×512 resolution image, the neural network backbone convolutionally adds 64×64 information and 32×32 information to the detection layer, and the five convolutional layers add 16×16 information, 8×8 information, 4×4 information, 2×2 information, and 1×1 information to the detection layer, respectively.

[0010] In some embodiments, the semantic segmentation module is a pyramid pooling module.

[0011] In some embodiments, the semantic segmentation module includes, in sequence: a pooling module, which pools the feature map to obtain pooled features of different sizes; a plurality of convolutional layers, each of which convolves one of the pooled features of different sizes to obtain convolutional features of different sizes; an upsampling module, which receives the convolutional features of different sizes to obtain upsampled features; a splicing layer, which receives the upsampled features and features from the neural network backbone to obtain spliced ​​features; and a convolutional layer, which convolves the spliced ​​features to obtain a pixel-by-pixel prediction as the semantic segmentation result. In some embodiments, the sizes of the pooled features are 1×1, 2×2, 3×3, and 6×6.

[0012] In some embodiments, the computer executable code is further configured to control an operating device in the scene based on the object detection result and the semantic segmentation result.

[0013] In some aspects, the present disclosure relates to a method for object detection and semantic segmentation. In some embodiments, the method includes: receiving, via a computing device, an image of a scene; processing, via the computing device, the image using a neural network backbone to obtain a feature map; processing, via the computing device, the feature map using an object detection module to obtain an object detection result for the image; and processing, via the computing device, the feature map using a semantic segmentation module to obtain a semantic segmentation result for the image, wherein the object detection module and the semantic segmentation module are trained using the same loss function, the loss function including an object detection component and a semantic segmentation component.

[0014] In some embodiments, the training data for the object detection module and the semantic segmentation module includes: a training image, at least one bounding box defined in the training image, a label of the at least one bounding box, and a mask of the training image.

[0015] In some embodiments, the neural network backbone comprises a ResNet18 backbone truncated from its 4th block.

[0016] In some embodiments, the object detection module is a single shot detector (SSD).

[0017] In some embodiments, the object detection module includes, in sequence: five convolutional layers; a detection layer; and a non-maximum suppression (NMS) layer. In some embodiments, for a 512×512 resolution image, the neural network backbone convolutionally adds 64×64 information and 32×32 information to the detection layer, and the five convolutional layers add 16×16 information, 8×8 information, 4×4 information, 2×2 information, and 1×1 information to the detection layer, respectively.

[0018] In some embodiments, the semantic segmentation module is a pyramid pooling module.

[0019] In some embodiments, the semantic segmentation module includes, in sequence: a pooling module, which pools the feature map to obtain pooled features of different sizes; multiple convolution layers, each convolution layer convolves one of the pooled features of different sizes to obtain convolution features of different sizes; an upsampling module, which receives the convolution features of different sizes to obtain upsampled features; a splicing layer, which receives the upsampled features and features from the neural network backbone to obtain spliced ​​features; and a convolution layer, which convolves the spliced ​​features to obtain the semantic segmentation result.

[0020] In some embodiments, the method further includes: controlling an operating device in the scene based on the object detection result and the semantic segmentation result.

[0021] In some aspects, the present disclosure relates to a non-transitory computer-readable medium having stored thereon computer-executable code, which, when executed by a processor of a computing device, is configured to implement the above method.

[0022] These and other aspects of the disclosure will become apparent from the following description of the preferred embodiments taken in conjunction with the following drawings and illustrations thereof, although changes and modifications may be effected therein without departing from the spirit and scope of the novel concepts of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present disclosure will be more fully understood from the detailed description and accompanying drawings. The accompanying drawings illustrate one or more embodiments of the present disclosure and, together with the written description, serve to explain the principles of the present disclosure. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0024] Figure 1The structure of a model for simultaneous object detection and semantic segmentation according to some embodiments of the present disclosure is schematically depicted.

[0025] Figure 2 An object detection and semantic segmentation system according to some embodiments of the present disclosure is schematically depicted.

[0026] Figure 3 The present invention schematically illustrates a method for training an object detection and semantic segmentation model according to some embodiments of the present disclosure.

[0027] Figure 4 Schematically depicts methods for using object detection and semantic segmentation models according to some embodiments of the present disclosure.

[0028] Figure 5 Schematically depicts an exemplary structure of an object detection and semantic segmentation model according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure will now be described in detail. With reference to the accompanying drawings, like numbers indicate like components throughout. In addition, some terms used in this specification are defined in more detail below.

[0030] The terms used in this specification generally have their ordinary meanings in the art, in the context of this disclosure, and in the specific context in which each term is used. Some of the terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance on the description of the present disclosure. It is understood that the same thing can be expressed in more than one way. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and there is no special significance as to whether a term is elaborated or discussed in detail herein. The present disclosure provides synonyms for some terms. The use of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any term discussed herein, is illustrative only and in no way limits the scope and meaning of the present disclosure or any exemplary term. Likewise, the present disclosure is not limited to the various embodiments given in this specification.

[0031] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. In the event of any conflict, the present document, including definitions, will control.

[0032] As used in the description and throughout the claims herein, the meanings of "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Furthermore, as used in the description and claims of the present disclosure, the meaning of "in" and "on" includes "in" and "on" unless the context clearly dictates otherwise. Furthermore, headings or subheadings may be used in the specification for the convenience of the reader, without affecting the scope of the present disclosure.

[0033] As used herein, "plurality" refers to two or more. As used herein, the terms "include," "comprising," "carrying," "having," "containing," "involving," etc. should be understood as open-ended, meaning including but not limited to.

[0034] As used herein, the phrase "at least one of A, B, and C" should be construed to mean a logical (A or B or C), using a non-exclusive logical OR. It should be understood that one or more steps within a method can be executed in different order (or concurrently) without altering the principles of the present disclosure.

[0035] As used herein, the term "module" may refer to, for example, a component that is or includes an Application Specific Integrated Circuit (ASIC) in a system-on-chip; an electronic circuit; a combinational logic circuit; a Field Programmable Gate Array (FPGA); a (shared or dedicated) processor or processor group that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above. The term "module" may also include a (shared or dedicated) memory or memory group that stores code executed by a processor.

[0036] The term "code" as used herein may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. The term "shared" used above refers to the ability to execute some or all of the code from multiple modules using a single (shared) processor. Additionally, some or all of the code from multiple modules may be stored in a single (shared) memory. The term "group" used above refers to the ability to execute some or all of the code from a single module using a processor group. Additionally, a memory group may be used to store some or all of the code from a single module.

[0037] As used herein, the term "interface" generally refers to a communication tool or device used to perform data communication between components at a point of interaction between components. Generally speaking, an interface can be applicable at both the hardware level and the software level, and can be a unidirectional interface or a bidirectional interface. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. The components that communicate with the interface can be, for example, multiple components of a computer system or peripheral devices.

[0038] The present disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components, which are shown as solid-line boxes, and virtual software components, which are shown as dashed-line boxes. Those skilled in the art will understand that, unless otherwise specified, these computer components may be implemented in the form of software, firmware, or hardware components, or a combination thereof, but are not limited to these forms.

[0039] The devices, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory, tangible, computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory, tangible, computer-readable media are nonvolatile memory, magnetic storage, and optical storage.

[0040] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0041] In some aspects, the present disclosure relates to a computationally efficient framework for simultaneous object detection and semantic segmentation in real time. The framework is a novel lightweight deep neural network. Figure 1 As shown, framework 100 includes a neural network backbone 104, an object detection module 106, and a semantic segmentation module 108. Object detection module 106 and semantic segmentation module 108 share the same neural network backbone 104. When an image of scene 102 is used as input, neural network backbone 104 processes the image to obtain a feature map. The feature map is used as input to object detection module 106 and semantic segmentation module 108. Object detection module 106 processes the feature map to obtain an image 110, in which detected objects are defined using bounding boxes in image 110 and labeled with the names of the objects. Semantic segmentation module 108 processes the feature map to obtain a mask 112, in which pixels in the image are classified into categories.

[0042] The object detection module 106 and the semantic segmentation module 108 are pre-trained before use, and a hybrid loss function is defined for both the object detection module 106 and the semantic segmentation module 108 during training. By defining the hybrid loss function, the training of the object detection module 106 and the semantic segmentation module 108 is faster and more accurate.

[0043] Furthermore, due to the shared neural network backbone 104 and the simple structure of the object detection module 106 and semantic segmentation module 108, the entire trained system operates very quickly and requires relatively low computing power. Therefore, real-time object detection and semantic segmentation can be achieved on low-computing machines such as the NVIDIA Jetson TX2. Furthermore, through a multi-task network, the two tasks can mutually enhance each other's accuracy.

[0044] Figure 2 Schematically illustrates a system for simultaneous real-time object detection and semantic segmentation according to some embodiments of the present disclosure. Figure 2 As shown, system 200 includes a computing device 210, an image acquisition device 250, and an operating device 270. Image acquisition device 250 and operating device 270 communicate with computing device 210 via, for example, the same or different system interfaces or network 240. In some embodiments, computing device 210 can be a general-purpose computer, a dedicated computer, a smartphone, a tablet, a controller, or an embedded device such as in an autonomous vehicle. Computing device 210 provides simultaneous object detection and semantic segmentation services based on the same image or the same set of images.

[0045] In some embodiments, the system interface 240 can be various buses, storage devices, other I / O devices, or USB interfaces. When the interface is a network, the network 240 can be various forms of wired networks or wireless networks. Examples of networks can include, but are not limited to, local area networks (LANs), wide area networks (WANs) including the Internet, or any other type of network. In some embodiments, the system interface 240 can be any other type of interface that communicatively connects the computing device 210 to the image acquisition device 250 and the operating device 270.

[0046] In some embodiments, the image acquisition device 250 can be a camera, a grayscale camera, an RGB camera, an RGB depth camera, a depth camera, a LIDAR, or a combination thereof. The image acquisition device 250 is used to capture images of an environment or scene, preferably continuously in the form of a video.

[0047] In some embodiments, operating device 270 may be an autonomous vehicle or any other device that requires instructions based on object detection and semantic segmentation information. When computing device 210 receives images or videos from image acquisition device 250 and performs real-time object detection and semantic segmentation, computing device 210 may use the obtained real-time information to instruct the operation of operating device 270. In some embodiments, operating device 270 may be controlled by a controller external to computing device 210, which is configured to receive object detection and semantic segmentation information from computing device 210 and use this information to instruct operating device 270 to operate.

[0048] In some embodiments, as Figure 2 As shown, computing device 210 may include, but is not limited to, a processor 212, a memory 214, and a storage device 216. In some embodiments, computing device 210 may include other hardware components and software components (not shown) to perform corresponding tasks. Examples of these hardware components and software components include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0049] The processor 212 may be a central processing unit (CPU) that controls the operation of the computing device 210. The processor 212 may execute an operating system (OS) or other applications of the computing device 210. In some embodiments, the server computing device 210 may have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs.

[0050] The memory 214 may be a volatile memory, such as a random-access memory (RAM), for storing data and information during operation of the computing device 210. In some embodiments, the memory 214 may be a volatile memory array. In some embodiments, the computing device 210 may operate on more than one memory 214.

[0051] Storage device 216 is a non-volatile data storage medium used to store an operating system (not shown) and other applications of inventory allocation computing device 210. Examples of storage device 216 include non-volatile memory such as flash memory, memory cards, USB drives, hard drives, floppy disks, optical drives, or any other type of data storage device. In some embodiments, computing device 210 may have multiple storage devices 216. These storage devices 216 may be the same storage device or different types of storage devices. Applications of computing device 210 may be stored in one or more storage devices 216 of computing device 210.

[0052] The storage device 216 includes an object detection and semantic segmentation application 218 and (optionally) training data 232 and a control application 234. The object detection and semantic segmentation application 218 is used to process the collected images using an object detection model and a semantic segmentation model to identify objects in the collected images and classify pixels in the collected images. The object detection and semantic segmentation application 218 is trained using the training data 232. In some embodiments, the object detection model and the semantic segmentation model of the object detection and semantic segmentation application 218 can be pre-trained before being installed on the storage device 216, and the storage device 216 does not need to have the training data 236. In some embodiments, the storage device 216 may include other modules required for training and performing semantic segmentation. The control application 234 is used to control the operation of the operating device 270 based on the object detection results and semantic segmentation results received from the object detection and semantic segmentation application 218 and the operating instructions corresponding to the results. The operating instructions and their corresponding relationship with the results can be pre-stored in the control application 234.

[0053] like Figure 2 As shown, the object detection and semantic segmentation application 218 includes an image acquisition module 220, a neural network backbone 222, an object detection module 224, a semantic segmentation module 226, a training module 228, and a hybrid loss function module 230. In some embodiments, the object detection and semantic segmentation application 218 is executed in real time, and the recognized objects and classified pixels are used in real time to control the operating device 270. In some embodiments, the object detection and semantic segmentation application 218 may not include one or more of the image acquisition module 220, the training module 228, and the hybrid loss function module 230, but may include at least the neural network backbone 222, the object detection module 224, and the semantic segmentation module 226. The neural network backbone 222, the object detection module 224, and the semantic segmentation module 226 are preferably pre-trained using a hybrid loss function that takes into account both object detection and semantic segmentation, wherein the hybrid loss function is defined by the hybrid loss function module 230.

[0054] The image acquisition module 220 is used to instruct the image acquisition device 250 to capture images or videos of a scene or environment in real time. In some embodiments, the image acquisition module 220 may also passively receive images captured by the image acquisition device 250. After the image acquisition module 220 obtains the image, it may further pre-process the image. Pre-processing may include adjusting the light balance of the image, reformatting the image, adjusting the image size, and (optionally) synchronizing images from different resources (e.g., multiple cameras of the image acquisition device 250). The image acquisition module 220 then sends the processed image to the neural network backbone 222.

[0055] The neural network backbone 222 is used to process one or more images after receiving them from the image acquisition module 220 to obtain a feature map. In some embodiments, the neural network backbone 222 is a ResNet18 backbone. In some embodiments, the feature map obtained includes a plurality of pixels, each pixel having a value or vector representing a feature of the pixel. Each vector has a dimension of length or C, and the value of C can be learned during the training of the application 218. In one example, the size of the input image is 512×512 (height H and width W), and after convolution by the neural network backbone 222, the output may include a 64×64 feature map and a 32×32 feature map. The neural network backbone 222 is used to send the 64×64 feature map and the 32×32 feature map to the object detection module 224, and to send the 64×64 feature map to the semantic segmentation module 226.

[0056] Object detection module 224 is configured to process the feature map received from neural network backbone 222 to identify objects in an image. In some embodiments, object detection module 224 may use a single shot detector (SSD), a deconvolutional single shot detector (DSSD), a light-head region-convolutional neural network (R-CNN), a Faster R-CNN, a region-based fully convolutional network (R-FCN), a feature pyramid network (FPN), RetinaNet, or You Only Look Once (Yolo) for object detection. In one example, a single shot detector (SSD) is used for object detection.

[0057] After receiving the feature map from the neural network backbone 222, the semantic segmentation module 226 processes the feature map to classify the pixels of the image. In some embodiments, the semantic segmentation module 226 may use a Pyramid Scene Parsing Network (PSPNet) or an FCN for semantic segmentation. In one example, PSPNet is used for semantic segmentation.

[0058] The training module 228 is used to train the neural network backbone 222 , the object detection module 224 , and the semantic segmentation module 226 using the training data 232 and the hybrid loss function provided by the hybrid loss function module 230 .

[0059] Hybrid loss function module 230 is used to provide a hybrid loss function for training neural network backbone 222, object detection module 224, and semantic segmentation module 226. The hybrid loss function includes components for both object detection and semantic segmentation. By providing the hybrid loss function and training data 222, the training process becomes more efficient because object detection and semantic segmentation can promote each other.

[0060] The training data 232 includes data for training both object detection and semantic segmentation. Specifically, for each data entry in the training data 232, the training data 232 provides an image, a bounding box of an object in the image, a label of the object in the bounding box (e.g., the name of the object), and a class label for the pixels in the image.

[0061] In some aspects, the present disclosure relates to a method for training an object detection and semantic segmentation application 218, and more particularly to a method for training a neural network backbone 222, an object detection module 224, and a semantic segmentation module 226. In some embodiments, as Figure 3 The method shown can be used in Figure 2 It should be noted that, unless otherwise specified in the present disclosure, the steps of the method can be arranged in a different order, and are therefore not limited to Figure 3 in the order shown.

[0062] like Figure 3 As shown, in step 302, the training module 228 retrieves the training data 232 and sends the training data to the neural network backbone 222. Each training entry of the training data 232 includes an image, a bounding box of an object in the image, a label of the object in the bounding box, and a class label for each pixel in the image.

[0063] In step 304, after receiving training data 232, neural network backbone 222 converts the image into a feature map and sends the feature map to object detection module 224 and semantic segmentation module 226, respectively. In performing step 304, the image is used without the bounding box, bounding box / object label, and pixel label of the image. The bounding box and label of the training image can be used later.

[0064] In step 306, after receiving the feature map from neural network backbone 222, object detection module 224 detects objects based on the feature map. Detection results may include bounding boxes in the image and labels indicating the bounding boxes of different objects. Labels may include, for example, cars, bicycles, people, traffic lights, etc.

[0065] In step 308, after receiving the feature map from the neural network backbone 222, the semantic segmentation module 226 segments the image and provides a class label for each pixel based on the feature map. The label for each pixel may include road, person, sidewalk, building, car, etc.

[0066] In step 310, after object detection by the object detection module 224 and semantic segmentation by the semantic segmentation module 226 are completed, the training module 228 compares the object detection results and the semantic segmentation results with the original labels for the training images to obtain the difference between the predicted results and the labels in the training data, retrieves the loss function provided by the hybrid loss function module 230, and uses the loss function and the difference to adjust the parameters or weights of the neural network backbone 222, the object detection module 224 and the semantic segmentation module 226.

[0067] In step 312 , steps 302 to 310 are repeated several times until the parameters or weights of the neural network backbone 222 , the object detection module 224 , and the semantic segmentation module 226 converge.

[0068] In some aspects, the present disclosure relates to a Figure 3 The method of using the well-trained application 218 described in. Figure 4 Schematically depicts a method for simultaneous real-time object detection and semantic segmentation according to some embodiments of the present disclosure. In some embodiments, as Figure 4 The method shown can be used in Figure 2 It should be noted that, unless otherwise specified in the present disclosure, the steps of the method can be arranged in a different order, and are therefore not limited to Figure 4 In some embodiments, Figure 4 The method shown is similar to Figure 3The difference of the method shown is that the input image is a real-time acquired image instead of a labeled training image, and there is no need to compare the predicted object detection results and semantic segmentation results.

[0069] like Figure 4 As shown, in step 402, the image acquisition device 250 captures an image of the scene and sends the image to the image acquisition module 220. The image acquisition module 220 receives the image and sends the image to the neural network backbone 222. In some embodiments, the image acquisition module 220 may pre-process the image before sending it for further processing, such as reformatting the image or resizing the image to meet the requirements of the neural network backbone 222.

[0070] In step 404, after receiving the collected image, the neural network backbone 222 converts the collected image into a feature map and sends the feature map to the object detection module 224 and the semantic segmentation module 226 respectively.

[0071] In step 406, after receiving the feature map from neural network backbone 222, object detection module 224 detects objects based on the feature map. Detection results may include bounding boxes in the captured image and labels indicating the bounding boxes of different objects. Labels may include, for example, cars, bicycles, people, traffic lights, etc.

[0072] In step 408, the semantic segmentation module 226 segments the image upon receiving the feature map from the neural network backbone 222 and provides a class label for each pixel based on the feature map. The label for each pixel may include road, person, sidewalk, building, car, etc.

[0073] In some embodiments, the method may further include step 410. In step 410, the control application 234 uses the object detection results from the object detection module 224 and the semantic segmentation results from the semantic segmentation module 226 and instructs the operation of the operating device 270 based on these results.

[0074] Figure 5 An example of an object detection and semantic segmentation model according to some embodiments of the present disclosure is shown. In some embodiments, the model corresponds to a neural network backbone 222, an object detection module 224, and a semantic segmentation module 226, and is further defined using a hybrid loss function that includes both an object detection component and a semantic segmentation component.

[0075] like Figure 5As shown, model 500 includes a shared ResNet18 backbone, an object detection sub-model, and a semantic segmentation sub-model. The shared ResNet18 backbone converts the image into a feature map. The feature map is input to the object detection sub-model and the semantic segmentation sub-model. In some embodiments, the object detection sub-model is implemented using SSD, and the semantic segmentation sub-model is implemented using pyramid pooling.

[0076] The object detection sub-model processes feature maps from the ResNet18 backbone using multiple convolutional layers of varying sizes. The results from the convolutional layers are combined with information directly from the ResNet18 backbone to perform object detection. This detection is further processed using non-maximum suppression (NMS) to obtain the final object detection result.

[0077] The semantic segmentation sub-model pools feature maps of different sizes and performs convolution on feature maps of different H and W sizes. After convolution, the semantic segmentation sub-model upsamples the convolution results to the same H and W sizes and concatenates the upsampled feature maps to produce a concatenated feature map. The concatenated feature map is then convolved to obtain the final semantic segmentation result. The result predicts a category for each pixel in the captured image. These categories can include ground, people, buildings, utility poles, and trees.

[0078] In some embodiments, the model has Figure 5 In some embodiments, the model has the following Figure 5 The exact hierarchical structure shown is not shown, but the sizes of dimensions H and W can be varied or changed proportionally (keeping dimension C the same). In some embodiments, the model has Figure 5 The exact hierarchical structure shown is not shown, while the sizes of dimensions H and W can be varied or changed proportionally (dimension C can be changed but not proportionally to the changes in H and W). In some embodiments, the model has Figure 5 The exact hierarchical structure shown is not shown, but the sizes of dimensions H, W, and C may vary or change proportionally. In some embodiments, Figure 5 The sizes of the layers in the shown model are for illustration only and are not intended to limit the scope of the present disclosure. Depending on the type of data to be processed, various sizes of dimensions H, W, C can be used.

[0079] In some aspects, the present disclosure relates to a non-transitory computer-readable medium having stored thereon computer-executable code. In some embodiments, the computer-executable code may be software stored in the non-volatile storage device 216 described above. When executed, the computer-executable code may perform one of the methods described above. In some embodiments, the non-transitory computer-readable medium may include, but is not limited to, the storage device 216 of the computing device 210 described above, or any other storage medium of the computing device 210.

[0080] In addition, some embodiments of the present disclosure provide simultaneous real-time object detection and semantic segmentation. Advantages of some embodiments of the present disclosure are as follows: (1) A shared lightweight backbone network makes the model faster; (2) The interaction between SSD and feature pyramid pooling is performed through at least a shared ResNet18 backbone and a hybrid loss function, making prediction more efficient and accurate; (3) Simultaneous object detection and segmentation helps control operating devices such as autonomous vehicles.

[0081] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0082] The embodiments are chosen and described in order to explain the principles of the present disclosure and its practical application, thereby enabling others skilled in the art to utilize the present disclosure and the various embodiments and various modifications as are suitable for the specific use contemplated. Alternative embodiments will become apparent to those skilled in the art without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Claims

1. An object detection and semantic segmentation system, the system comprising a computing device comprising a processor and a non-volatile memory storing computer executable code, wherein: The computer executable code, when executed at the processor, is configured to: receiving an image of a scene; processing the image using a neural network backbone to obtain a feature map; Processing the feature map using an object detection module to obtain an object detection result for the image; as well as Processing the feature map using a semantic segmentation module to obtain a semantic segmentation result of the image, wherein the neural network backbone, the object detection module, and the semantic segmentation module are trained using the same loss function, the loss function including an object detection component and a semantic segmentation component; and wherein the neural network backbone comprises a ResNet18 backbone, and the ResNet18 backbone is truncated from the fourth block thereof; The object detection module includes: Five convolutional layers; Detection layer; and Non-maximum suppression NMS layer; The using an object detection module to process the feature map to obtain an object detection result of the image includes: Processing the feature map from the ResNet18 backbone using the five convolutional layers; The non-maximum suppression (NMS) layer is used to process the information combining the results from each of the five convolutional layers with the feature map from the ResNet18 backbone to obtain the detection result of the object.

2. The system according to claim 1, wherein: The training data for the object detection module and the semantic segmentation module includes: a training image, at least one bounding box defined in the training image, a label of the at least one bounding box, and a mask of the training image.

3. The system according to claim 1, wherein: The object detection module is a single shot detector SSD.

4. The system according to claim 1, wherein: For an image with a resolution of 512×512, the neural network backbone convolutionally adds 64×64 information and 32×32 information to the detection layer, and the five convolutional layers add 16×16 information, 8×8 information, 4×4 information, 2×2 information and 1×1 information to the detection layer respectively.

5. The system according to claim 1, wherein The semantic segmentation module is a pyramid pooling module.

6. The system according to claim 1, wherein: The semantic segmentation module includes: A pooling module, wherein the pooling module pools the feature map to obtain pooled features of different sizes; A plurality of convolutional layers, each convolutional layer convolves one of the pooled features of different sizes to obtain convolutional features of different sizes; an upsampling module, wherein the upsampling module receives the convolution features of different sizes to obtain upsampling features; a concatenation layer receiving the upsampled features and features from the neural network backbone to obtain concatenated features; and A convolutional layer convolves the concatenated features to obtain a pixel-by-pixel prediction as the semantic segmentation result.

7. The system according to claim 6, wherein: The sizes of the pooled features are 1×1, 2×2, 3×3 and 6×6.

8. The system according to claim 1, wherein: The computer executable code is further configured to control an operating device in the scene based on the object detection result and the semantic segmentation result.

9. A method for object detection and semantic segmentation, comprising: receiving, via a computing device, an image of a scene; processing the image using a neural network backbone by the computing device to obtain a feature map; Processing the feature map using an object detection module by the computing device to obtain an object detection result for the image; as well as Processing the feature map using a semantic segmentation module by the computing device to obtain a semantic segmentation result of the image, wherein the neural network backbone, the object detection module, and the semantic segmentation module are trained using the same loss function, the loss function including an object detection component and a semantic segmentation component; and wherein the neural network backbone comprises a ResNet18 backbone, and the ResNet18 backbone is truncated from the fourth block thereof; The object detection module includes: Five convolutional layers; Detection layer; and Non-maximum suppression NMS layer; The using an object detection module to process the feature map to obtain an object detection result of the image includes: Processing the feature map from the ResNet18 backbone using the five convolutional layers; The non-maximum suppression (NMS) layer is used to process the information combining the results from each of the five convolutional layers with the feature map from the ResNet18 backbone to obtain the detection result of the object.

10. The method according to claim 9, wherein: The training data for the object detection module and the semantic segmentation module includes: a training image, at least one bounding box defined in the training image, a label of the at least one bounding box, and a mask of the training image.

11. The method according to claim 9, wherein The object detection module is a single-shot detector.

12. The method according to claim 9, wherein For an image with a resolution of 512×512, the neural network backbone convolutionally adds 64×64 information and 32×32 information to the detection layer, and the five convolutional layers add 16×16 information, 8×8 information, 4×4 information, 2×2 information and 1×1 information to the detection layer respectively.

13. The method according to claim 9, wherein: The semantic segmentation module is a pyramid pooling module.

14. The method according to claim 9, wherein The semantic segmentation module includes: A pooling module, wherein the pooling module pools the feature map to obtain pooled features of different sizes; A plurality of convolutional layers, each convolutional layer convolves one of the pooled features of different sizes to obtain convolutional features of different sizes; an upsampling module, wherein the upsampling module receives the convolution features of different sizes to obtain upsampling features; a concatenation layer receiving the upsampled features and features from the neural network backbone to obtain concatenated features; and A convolution layer is used to convolve the concatenated features to obtain the semantic segmentation result.

15. The method according to claim 9, further comprising: An operating device in the scene is controlled based on the object detection result and the semantic segmentation result.

16. A non-transitory computer-readable medium storing computer-executable code, wherein: The computer executable code, when executed at a processor of a computing device, is configured to: receiving an image of a scene; processing the image using a neural network backbone to obtain a feature map; Processing the feature map using an object detection module to obtain an object detection result for the image; as well as Processing the feature map using a semantic segmentation module to obtain a semantic segmentation result of the image, wherein the neural network backbone, the object detection module, and the semantic segmentation module are trained using the same loss function, the loss function including an object detection component and a semantic segmentation component; and wherein the neural network backbone comprises a ResNet18 backbone, and the ResNet18 backbone is truncated from the fourth block thereof; The object detection module includes: Five convolutional layers; Detection layer; and Non-maximum suppression NMS layer; The using an object detection module to process the feature map to obtain an object detection result of the image includes: Processing the feature map from the ResNet18 backbone using the five convolutional layers; The non-maximum suppression (NMS) layer is used to process the information combining the results from each of the five convolutional layers with the feature map from the ResNet18 backbone to obtain the detection result of the object.

17. The non-transitory computer-readable medium of claim 16, wherein: The semantic segmentation module includes: A pooling module, wherein the pooling module pools the feature map to obtain pooled features of different sizes; A plurality of convolutional layers, each convolutional layer convolves one of the pooled features of different sizes to obtain convolutional features of different sizes; an upsampling module, wherein the upsampling module receives the convolution features of different sizes to obtain upsampling features; a concatenation layer receiving the upsampled features and features from the neural network backbone to obtain concatenated features; and A convolutional layer convolves the concatenated features to obtain a pixel-by-pixel prediction as the semantic segmentation result.

Images