Semantic segmentation model training method, semantic segmentation method, device and electronic device
By combining the depth estimation network and 3D point cloud segmentation in semantic segmentation model training, and using the consistency loss function to update the model parameters, the problem of insufficient accuracy of 2D segmentation in semantic segmentation model training in 3D visual scenes in the prior art is solved, and the problem of insufficient accuracy of 2D segmentation in semantic segmentation model training in 3D visual scenes is realized. Higher accuracy and robustness.
Patent Information
- Application Number
- CN202110724394.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the training of semantic segmentation model in 3D visual scenes, it is difficult to effectively utilize 3D information to improve the accuracy and robustness of 2D semantic segmentation.
Through the depth estimation network and semantic segmentation network based on the initial semantic segmentation model, the depth map and two-dimensional segmentation result map of the sample picture are determined, and then converted to 3D point cloud based on the depth map, and semantic segmentation is performed for 3D point clouds, combining two-dimensional The consistency loss function of the three-dimensional segmentation result updates the model parameters until converges, and obtains the target semantic segmentation model.
The accuracy and robustness of the semantic segmentation model are improved, and the 3D segmentation results are introduced as consistent as possible with the 2D segmentation results, which improves the performance of the training model.
Smart Images

Figure CN113409340B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision and deep learning technology, and can be specifically used in 3D vision scenarios. Background Art
[0002] In the field of computer vision, the current applications of neural networks mainly include image recognition, object localization and detection, and semantic segmentation. Image recognition tells you what the image is, object localization and detection tells you where the object is in the image, and semantic segmentation answers the above two questions at the pixel level.
[0003] Image semantic segmentation, literally means letting the computer segment the image based on its semantics. Figure 1 In the case of the left image, the right image can be output. In speech recognition, semantics refers to the meaning of speech. In the field of images, semantics refers to the content of the image, the understanding of the meaning of the image. For example, the semantics of the left image is three people riding three bicycles. Segmentation means segmenting different objects in the image from the perspective of pixels, and labeling each pixel in the original image, such as Figure 1 The middle right picture corresponds to the human part and the bicycle part respectively. Summary of the invention
[0004] The present disclosure provides a semantic segmentation model training method, a semantic segmentation method, a device and an electronic device.
[0005] According to a first aspect of the present disclosure, a semantic segmentation model training method is provided, comprising:
[0006] Based on the depth estimation network and semantic segmentation network of the initial semantic segmentation model, the depth map and two-dimensional segmentation result map of the sample image are determined;
[0007] Determine the 3D point cloud based on the depth map of the sample image, and perform semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image;
[0008] Determine a loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, wherein the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map;
[0009] Based on the determined loss function value, the model parameters of the initial semantic segmentation model are updated until convergence to obtain the target semantic segmentation model.
[0010] According to a second aspect of the present disclosure, a semantic segmentation method is provided, comprising:
[0011] Determine the target image to be segmented;
[0012] The target image to be segmented is input into the target semantic segmentation model trained in the first aspect to obtain the semantic segmentation result of the target image to be segmented.
[0013] According to a third aspect of the present disclosure, a semantic segmentation model training device is provided, comprising:
[0014] A first determination module is used to determine a depth map and a two-dimensional segmentation result map of a sample image based on a depth estimation network and a semantic segmentation network of an initial semantic segmentation model;
[0015] A 3D semantic segmentation module is used to determine a 3D point cloud based on a depth map of a sample image, and to perform semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image;
[0016] A loss value determination module, used to determine a loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, wherein the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map;
[0017] The updating module is used to update the model parameters of the initial semantic segmentation model based on the determined loss function value until convergence to obtain the target semantic segmentation model.
[0018] According to a fourth aspect of the present disclosure, a semantic segmentation device is provided, comprising:
[0019] A second determination module is used to determine the target image to be segmented;
[0020] The module is used to input the target image to be segmented into the target semantic segmentation model trained in the first aspect to obtain the semantic segmentation result of the target image to be segmented.
[0021] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:
[0022] at least one processor; and
[0023] A memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method.
[0025] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.
[0026] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above method when executed by a processor.
[0027] The beneficial effects of the technical solution provided by the present disclosure are:
[0028] Compared with the prior art, the solution provided by the embodiment of the present disclosure mainly improves the accuracy of semantic segmentation in terms of network improvement and multimodal feature fusion. The present disclosure determines the depth map and the two-dimensional segmentation result map of the sample image through a depth estimation network and a semantic segmentation network based on the initial semantic segmentation model; then determines the 3D point cloud based on the depth map of the sample image, and performs semantic segmentation on the 3D point cloud to obtain the three-dimensional segmentation result map of the sample image; then determines the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, and the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map; finally, based on the determined loss function value, the model parameters of the initial semantic segmentation model are updated until convergence to obtain the target semantic segmentation model. That is, when training the semantic segmentation model, by introducing the consistency loss function of 3D semantic segmentation and 2D semantic segmentation, the 3D segmentation result is as consistent as possible with the 2D segmentation result, thereby using 3D information to guide the 2D semantic segmentation, thereby improving the accuracy and robustness of the trained model.
[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0031] Figure 1 is an example diagram of semantic segmentation provided according to the present disclosure;
[0032] Figure 2 is a flow chart of a semantic segmentation model training method provided by the present disclosure;
[0033] Figure 3 is an example diagram of a semantic segmentation model training method provided in the present disclosure;
[0034] Figure 4 is a flowchart of a semantic segmentation method provided by the present disclosure;
[0035] Figure 5 A schematic diagram of the structure of the semantic segmentation model training device provided by the present disclosure;
[0036] Figure 6 A schematic diagram of the structure of the semantic segmentation device provided by the present disclosure;
[0037] Figure 7 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION
[0038] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0039] Embodiment 1
[0040] Figure 1 A semantic segmentation model training method provided by an embodiment of the present disclosure is shown. Figure 2 As shown, the method includes:
[0041] Step S101, determining a depth map and a two-dimensional segmentation result map of a sample image based on a depth estimation network and a semantic segmentation network of an initial semantic segmentation model;
[0042] Specifically, the sample image is input into the initial semantic segmentation network model, and the depth map and the two-dimensional segmentation result map of the sample image are determined through the depth estimation network and the semantic segmentation network of the initial semantic segmentation model.
[0043] Among them, depth estimation is to use RGB images to estimate the distance of each pixel in the image relative to the shooting source. When taking a photo, the three-dimensional graphics are projected onto a two-dimensional plane to form a two-dimensional image. The purpose of depth estimation is to estimate three-dimensional information through a two-dimensional image, which is an inverse process. Depth estimation (estimating depth from a 2D image) is a key step in scene reconstruction and understanding tasks, and is part of 3D reconstruction in the field of computer vision. The basis of monocular estimation based on deep learning is that the relationship between pixel values reflects the depth relationship. The method is to fit a function to map the image into a depth map. Obtaining a specific depth from a single picture is equivalent to inferring a three-dimensional space from a two-dimensional image. Depth estimation can be divided into monocular depth estimation and multi-eye depth estimation.
[0044] Specifically, the depth estimation network can be a depth estimation network based on a convolutional neural network, such as the network introduced in the paper "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network", or it can be other networks that can realize the functions of this application.
[0045] Among them, the semantic segmentation of the image is to assign a semantic category to each pixel in the input image to obtain a pixel-based dense classification.
[0046] Specifically, the semantic segmentation network can be a network based on fully convolutional networks (FCNs), SegNet, E-Net, Link-Net, Mask R-CNN, or other networks that can realize the functions of this application, and this application is not limited here.
[0047] Specifically, the sample set of the present application can be determined by manual labeling, or the sample set can be obtained by processing unlabeled sample data in an unsupervised or weakly supervised manner. The training sample set can include positive samples and negative samples.
[0048] Step S102, determining a 3D point cloud based on the depth map of the sample image, and performing semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image;
[0049] Among them, point cloud data refers to a set of vectors in a three-dimensional coordinate system. 3D data can usually be represented in different formats, including depth images, point clouds, meshes, and volume meshes. As a common format, point cloud representation retains the original geometric information in 3D space without any discretization. Therefore, it is the preferred representation for many scene understanding related applications such as autonomous driving and robotics. 3D point cloud segmentation requires understanding of the global geometric structure and fine-grained details of each point. According to the segmentation granularity, 3D point cloud segmentation methods can be divided into three categories: semantic segmentation (scene level), instance segmentation (object level), and part segmentation (part level). Given a point cloud, the goal of semantic segmentation is to divide the point cloud into several subsets according to their semantics. There are two paradigms for semantic segmentation, namely projection-based methods and point-based methods.
[0050] Specifically, the 3D point cloud segmentation network of the present application can be a projection-based network or a point-based network, as well as other networks that can realize the functions of the present application.
[0051] Step S103, determining a loss function value based on the two-dimensional segmentation result image, the three-dimensional segmentation result image, and a predetermined loss function, wherein the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result image and the three-dimensional segmentation result image;
[0052] Specifically, the consistency loss function value may be determined based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, the two-dimensional segmentation and three-dimensional segmentation labels of the sample image, and a predetermined loss function.
[0053] Step S104, updating the model parameters of the initial semantic segmentation model based on the determined loss function value until convergence, thereby obtaining a target semantic segmentation model.
[0054] Specifically, back propagation can be performed in a direction that reduces the consistency loss, and the parameters of the model can be adjusted until convergence to obtain the target semantic segmentation model.
[0055] The present disclosure provides a possible implementation method, in which the consistency loss function is:
[0056]
[0057] in, Indicates the 2D semantic segmentation result corresponding to the pixel points of the sample image. Indicates the 3D semantic segmentation results corresponding to the pixels of the sample image.
[0058] For the embodiments of the present disclosure, the consistency loss function is used to guide two-dimensional semantic segmentation using 3D information.
[0059] The embodiment of the present disclosure provides a possible implementation method, wherein, based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, determining the loss function value includes:
[0060] Determine the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, the depth map and the two-dimensional segmentation label, the three-dimensional segmentation label, the depth information label of the sample image, and a predetermined loss function; wherein the predetermined loss function also includes: a two-dimensional loss function, a three-dimensional loss function, and a depth loss function.
[0061] For the disclosed embodiment, four loss functions are considered, namely: two-dimensional loss function, three-dimensional loss function, depth loss function, and consistency loss function, so as to further improve the accuracy and robustness of the trained model. In addition, certain weight values can be set for the three loss functions, where the weight values can be empirical values or obtained through training.
[0062] Exemplarily, the loss function of the semantic segmentation model of the present disclosure may be:
[0063] L all =L 2D-seg +L depth +L 3D-seg +L consist
[0064] Among them, the four loss functions represent two-dimensional loss function, three-dimensional loss function, depth loss function, and consistency loss function.
[0065] The present disclosure provides a possible implementation method, wherein determining a 3D point cloud based on a depth map of a sample image includes:
[0066] Based on the intrinsic parameters of the camera corresponding to the sample image and the depth map, the depth map is converted into a 3D point cloud.
[0067] The conversion of depth map to point cloud is actually a transformation of coordinate system, image coordinate system -> world coordinate system, and the constraint of the transformation is the camera intrinsic parameter. That is, based on the camera intrinsic parameter, the depth map can be converted into 3D point cloud, thus solving the problem of how to transform the depth map into point cloud.
[0068] The present disclosure provides a possible implementation method, in which, based on the intrinsic parameters of the camera corresponding to the sample image and the depth map, the depth map is converted into a 3D point cloud, including:
[0069] Through the following formula:
[0070]
[0071] Convert the depth map to a 3D point cloud; where d is the depth of each pixel (u, v), K is the intrinsic parameter of the camera, and P c is the 3D coordinate in the camera coordinate system [X c , Y c , Z c ].
[0072] Specifically, the embodiments of the present disclosure solve the problem of how to convert a depth map into point cloud data according to the above formula.
[0073] The embodiment of the present application provides a possible implementation method, wherein semantic segmentation is performed on a 3D point cloud to obtain a three-dimensional segmentation result map of a sample image, including:
[0074] Based on the predetermined 3D point cloud semantic segmentation network, the semantic segmentation result of the 3D point cloud is obtained.
[0075] Specifically, the point cloud segmentation network of the embodiment of the present disclosure may be Pinpoint, Pointnet++, etc., or may be other point cloud segmentation networks that can implement the present application. The embodiment of the present application implements semantic segmentation of point clouds.
[0076] Exemplarily, in order to better understand the training method of the semantic segmentation network model disclosed in the present invention, Figure 3 A training process example diagram is shown, including: 1. Encoding the input image, and then obtaining a depth map and a two-dimensional segmentation result based on the depth estimation network (Depth head) and the two-dimensional semantic segmentation network (seg head) of the initial model respectively; 3. Converting the depth map into a 3D point cloud; 4. Performing semantic segmentation on the 3D point cloud to obtain a 3D semantic segmentation network; 5. Determining the corresponding loss based on the consistency of 3D and 2D semantic segmentation, and adjusting the parameters of the initial model based on directional propagation in the direction where the loss value becomes smaller.
[0077] Embodiment 2
[0078] According to a second aspect of the present disclosure, an image recognition method is provided, such as Figure 4 Shown include:
[0079] Step S401, determining a target image to be segmented;
[0080] Step S402: input the target image to be segmented into the target semantic segmentation model trained in the first embodiment to obtain a semantic segmentation result of the target image to be segmented.
[0081] The target image to be segmented may be a picture directly taken by a camera, or may be extracted from a captured video.
[0082] As a scenario of an embodiment of the present disclosure, it may be a driving image captured by an on-board camera of an unmanned vehicle, and then each target in the driving image is determined, and the image is automatically segmented and classified to avoid obstacles such as pedestrians and vehicles.
[0083] As another application scenario of the embodiment of the present disclosure, it can be used for medical image analysis. With the rise of artificial intelligence, combining neural networks with medical diagnosis has also become a research hotspot. Intelligent medical research has gradually matured. In the field of intelligent medicine, semantic segmentation can be used for tumor image segmentation, caries diagnosis, etc.
[0084] In addition, the method disclosed in the present application can also be used in the security field, as well as other application scenarios involving target detection and segmentation of semantic segmentation.
[0085] For the embodiments of the present disclosure, the semantic segmentation model used is a model obtained by introducing 3D information to guide two-dimensional semantic segmentation training, which can improve the accuracy of semantic segmentation.
[0086] A possible implementation of an embodiment of the present application, wherein, the method further includes: inputting the target image to be segmented into a target semantic segmentation model to obtain a depth map of the target image to be segmented.
[0087] Specifically, in the embodiments of the present application, the depth information of the target image to be segmented can also be obtained through the trained semantic segmentation model.
[0088] For example, one application scenario of the depth information is that it can be used as reference data for vehicle obstacle avoidance during autonomous driving.
[0089] Embodiment 3
[0090] The present disclosure provides a semantic segmentation model training device. Figure 5 As shown, including:
[0091] A first determination module 501 is used to determine a depth map and a two-dimensional segmentation result map of a sample image based on a depth estimation network and a semantic segmentation network of an initial semantic segmentation model;
[0092] A 3D semantic segmentation module 502 is used to determine a 3D point cloud based on a depth map of a sample image, and perform semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image;
[0093] A loss value determination module 503, used to determine a loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, wherein the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map;
[0094] The updating module 504 is used to update the model parameters of the initial semantic segmentation model based on the determined loss function value until convergence to obtain the target semantic segmentation model.
[0095] This embodiment of the present application provides a possible implementation method, in which the consistency loss function is:
[0096]
[0097] in, Indicates the 2D semantic segmentation result corresponding to the pixel points of the sample image. Indicates the 3D semantic segmentation results corresponding to the pixels of the sample image.
[0098] An embodiment of the present application provides a possible implementation method, in which a loss value determination module is specifically used to determine a loss function value based on a two-dimensional segmentation result map, a three-dimensional segmentation result map, a depth map, and a two-dimensional segmentation label, a three-dimensional segmentation label, a depth information label of a sample image, and a predetermined loss function; wherein the predetermined loss function also includes: a two-dimensional loss function, a three-dimensional loss function, and a depth loss function.
[0099] The present application embodiment provides a possible implementation method, wherein the device further includes:
[0100] The conversion module is used to convert the depth map into a 3D point cloud based on the intrinsic parameters of the camera corresponding to the sample image and the depth map.
[0101] The embodiment of the present application provides a possible implementation method, wherein the conversion module is specifically used to use the following formula:
[0102]
[0103] Convert the depth map to a 3D point cloud; where d is the depth of each pixel (u, v), K is the intrinsic parameter of the camera, and P c is the 3D coordinate in the camera coordinate system [X c , Y c , Z c ].
[0104] An embodiment of the present application provides a possible implementation method, in which a 3D semantic segmentation module is specifically used to obtain a semantic segmentation result of a 3D point cloud based on a predetermined 3D point cloud semantic segmentation network.
[0105] For the embodiments of the present application, the beneficial effects achieved are the same as those of the above-mentioned method embodiments and will not be repeated here.
[0106] Embodiment 3
[0107] The present disclosure provides a semantic segmentation device, such as Figure 6 As shown, including:
[0108] The second determination module 601 is used to determine the target image to be segmented;
[0109] The obtaining module 602 is used to input the target image to be segmented into the target semantic segmentation model trained in the first embodiment to obtain the semantic segmentation result of the target image to be segmented.
[0110] The embodiment of the present application provides a possible implementation method, wherein the obtaining module is also used to input the target image to be segmented into the target semantic segmentation model of the first embodiment to obtain a depth map of the target image to be segmented.
[0111] For the embodiments of the present application, the beneficial effects achieved are the same as those of the above-mentioned method embodiments and will not be repeated here.
[0112] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0113] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0114] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the embodiment of the present disclosure.
[0115] Compared with the prior art, this electronic device mainly improves the accuracy of semantic segmentation in terms of network improvement and multimodal feature fusion. The present invention determines the depth map and two-dimensional segmentation result map of the sample image through a depth estimation network and a semantic segmentation network based on the initial semantic segmentation model; then determines the 3D point cloud based on the depth map of the sample image, and performs semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image; then determines the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, and the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map; finally, based on the determined loss function value, the model parameters of the initial semantic segmentation model are updated until convergence to obtain the target semantic segmentation model. That is, when training the semantic segmentation model, by introducing 3D and 2D consistency loss functions, the 3D segmentation result is made as consistent as possible with the 2D segmentation result, thereby using 3D information to guide 2D semantic segmentation, thereby improving the accuracy and robustness of the trained model.
[0116] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method provided by the embodiment of the present disclosure.
[0117] Compared with the prior art, the readable storage medium mainly improves the accuracy of semantic segmentation in terms of network improvement and multimodal feature fusion. The present disclosure determines the depth map and the two-dimensional segmentation result map of the sample image through a depth estimation network and a semantic segmentation network based on the initial semantic segmentation model; then determines the 3D point cloud based on the depth map of the sample image, and performs semantic segmentation on the 3D point cloud to obtain the three-dimensional segmentation result map of the sample image; then determines the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, and the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map; finally, based on the determined loss function value, the model parameters of the initial semantic segmentation model are updated until convergence to obtain the target semantic segmentation model. That is, when training the semantic segmentation model, by introducing the 3D and 2D consistency loss functions, the 3D segmentation result is made as consistent as possible with the 2D segmentation result, thereby using 3D information to guide the 2D semantic segmentation, thereby improving the accuracy and robustness of the trained model.
[0118] The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the method as shown in the first aspect of the present disclosure.
[0119] Compared with the prior art, this computer program product mainly improves the accuracy of semantic segmentation in terms of network improvement and multimodal feature fusion. The present invention determines the depth map and two-dimensional segmentation result map of the sample image through a depth estimation network and a semantic segmentation network based on the initial semantic segmentation model; then determines the 3D point cloud based on the depth map of the sample image, and performs semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image; then determines the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, and the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map; finally, based on the determined loss function value, the model parameters of the initial semantic segmentation model are updated until convergence to obtain the target semantic segmentation model. That is, when training the semantic segmentation model, by introducing 3D and 2D consistency loss functions, the 3D segmentation result is made as consistent as possible with the 2D segmentation result, thereby using 3D information to guide 2D semantic segmentation, thereby improving the accuracy and robustness of the trained model.
[0120] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0121] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0122] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0123] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as method semantic segmentation model training or semantic segmentation. For example, in some embodiments, the method semantic segmentation model training or semantic segmentation may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method semantic segmentation model training or semantic segmentation described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method semantic segmentation model training or semantic segmentation in any other suitable manner (eg, by means of firmware).
[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0125] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0128] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0131] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A semantic segmentation model training method, comprising: Based on the depth estimation network and semantic segmentation network of the initial semantic segmentation model, the depth map and two-dimensional segmentation result map of the sample image are determined; Determine a 3D point cloud based on the depth map of the sample image, and perform semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image; Based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, determining a loss function value, the predetermined loss function includes a consistency loss function, the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map, and the consistency loss function is obtained according to the 2D semantic segmentation result corresponding to the pixel points of the sample image and the 3D semantic segmentation result corresponding to the pixel points of the sample image; The determining of the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function includes: Determine the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, the depth map, the two-dimensional segmentation label, the three-dimensional segmentation label, the depth information label of the sample image, and a predetermined loss function; wherein the predetermined loss function also includes: a two-dimensional loss function, a three-dimensional loss function, and a depth loss function; The model parameters of the initial semantic segmentation model are updated based on the determined loss function value until convergence, thereby obtaining a target semantic segmentation model.
2. The method according to claim 1, wherein: The consistency loss function is: in, Indicates the 2D semantic segmentation result corresponding to the pixel points of the sample image, Indicates the 3D semantic segmentation result corresponding to the pixel points of the sample image.
3. The method according to claim 1, wherein: The determining of the 3D point cloud based on the depth map of the sample image includes: Based on the intrinsic parameters of the camera corresponding to the sample image and the depth map, the depth map is converted into a 3D point cloud.
4. The method according to claim 3, wherein: The converting the depth map into a 3D point cloud based on the intrinsic parameters of the camera corresponding to the sample image and the depth map includes: Through the following formula: Convert the depth map to a 3D point cloud; where d is the depth of each pixel (u, v), K is the intrinsic parameter of the camera, and P c is the 3D coordinate in the camera coordinate system [X c ,Y c ,Z c ].
5. The method according to claim 1, wherein: Performing semantic segmentation on the 3D point cloud to obtain a 3D segmentation result map of the sample image includes: Based on a predetermined 3D point cloud semantic segmentation network, a semantic segmentation result of the 3D point cloud is obtained.
6. A semantic segmentation method, comprising: Determine the target image to be segmented; The target image to be segmented is input into the target semantic segmentation model trained according to any one of claims 1 to 5 to obtain a semantic segmentation result of the target image to be segmented.
7. The method according to claim 6, wherein: The method also includes: inputting the target image to be segmented into the target semantic segmentation model according to any one of claims 1 to 5 to obtain a depth map of the target image to be segmented.
8. A semantic segmentation model training device, comprising: A first determination module is used to determine a depth map and a two-dimensional segmentation result map of a sample image based on a depth estimation network and a semantic segmentation network of an initial semantic segmentation model; A 3D semantic segmentation module, used to determine a 3D point cloud based on the depth map of the sample image, and perform semantic segmentation on the 3D point cloud to obtain a three-dimensional segmentation result map of the sample image; A loss value determination module, used to determine a loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, and a predetermined loss function, wherein the predetermined loss function includes a consistency loss function, and the consistency loss function is used to determine the consistency loss of the two-dimensional segmentation result map and the three-dimensional segmentation result map, and the consistency loss function is obtained according to the 2D semantic segmentation result corresponding to the pixel points of the sample image and the 3D semantic segmentation result corresponding to the pixel points of the sample image; The loss value determination module is specifically used to determine the loss function value based on the two-dimensional segmentation result map, the three-dimensional segmentation result map, the depth map and the two-dimensional segmentation label, the three-dimensional segmentation label, the depth information label of the sample image, and a predetermined loss function; wherein the predetermined loss function also includes: a two-dimensional loss function, a three-dimensional loss function, and a depth loss function; An updating module is used to update the model parameters of the initial semantic segmentation model based on the determined loss function value until convergence to obtain a target semantic segmentation model.
9. The device according to claim 8, wherein: The consistency loss function is: in, Indicates the 2D semantic segmentation result corresponding to the pixel points of the sample image, Indicates the 3D semantic segmentation result corresponding to the pixel points of the sample image.
10. The device according to claim 8, wherein: The device also includes: The conversion module is used to convert the depth map into a 3D point cloud based on the intrinsic parameters of the camera corresponding to the sample image and the depth map.
11. The device according to claim 10, wherein: The conversion module is specifically used to use the following formula: Convert the depth map to a 3D point cloud; where d is the depth of each pixel (u, v), K is the intrinsic parameter of the camera, and P c is the 3D coordinate in the camera coordinate system [X c ,Y c ,Z c ].
12. The device according to claim 8, wherein: The 3D semantic segmentation module is specifically used to obtain the semantic segmentation result of the 3D point cloud based on a predetermined 3D point cloud semantic segmentation network.
13. A semantic segmentation device, comprising: A second determination module is used to determine the target image to be segmented; An obtaining module is used to input the target image to be segmented into the target semantic segmentation model trained by any one of claims 1-5 to obtain a semantic segmentation result of the target image to be segmented.
14. The device according to claim 13, wherein: The obtaining module is also used to input the target image to be segmented into the target semantic segmentation model according to any one of claims 1 to 5 to obtain a depth map of the target image to be segmented.
15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.
17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Three-dimensional measurement method and apparatus, server, and storage medium
CN108895981A
Lightweight real-time semantic segmentation method based on three-dimensional point cloud
CN110570429A
Hip joint segmentation method and device, electronic equipment and storage medium
CN110648337A
Point cloud semantic segmentation method and system based on adversarial learning and multi-modal learning
CN112233124A
Method and device for three-dimensional image semantic segmentation, terminal and storage medium
WO2020078269A1
