Semantic segmentation method and apparatus, and computer-readable storage medium and robot

Abstract

The present application belongs to the technical field of robots, and in particular relates to a semantic segmentation method and apparatus, and a computer-readable storage medium and a robot. The method comprises: acquiring a first image and a second image, wherein the first image is an image to be subjected to semantic segmentation, and the second image is an image frame previous to the first image; performing optical-flow calculation on the first image and the second image, so as to obtain first optical-flow data; fusing the first optical-flow data with the first image, so as to obtain an optical-flow fusion image; and using a preset semantic segmentation model to perform semantic segmentation on the optical-flow fusion image, so as to obtain a semantic segmentation result corresponding to the first image. By means of the method, semantic segmentation can be performed in combination with optical-flow data of an image, such that deep features of the image can be mined, and semantic segmentation can be better performed on unknown objects, thereby effectively improving the accuracy of a semantic segmentation method.

Description

A semantic segmentation method, device, computer-readable storage medium and robot

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 28, 2023, with application number 202310781685.7 and invention name “A semantic segmentation method, device, computer-readable storage medium and robot”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of robotics technology, and in particular relates to a semantic segmentation method, device, computer-readable storage medium, and robot. Background Art

[0003] With the development of intelligent technology, semantic segmentation technology has been applied to various scenarios such as autonomous driving, indoor navigation, virtual reality and image recognition.

[0004] In existing technologies, before using a semantic segmentation model to segment an image, it is usually necessary to train the model using a semantic segmentation training set. However, when the scene changes, unknown objects that are not included in the semantic segmentation training set may appear in the new scene, resulting in low accuracy of the semantic segmentation method. Technical issues

[0005] In view of this, embodiments of the present application provide a semantic segmentation method, device, computer-readable storage medium, and robot to solve the problem of low accuracy of existing semantic segmentation methods when facing new scenarios. Technical Solutions

[0006] A first aspect of the embodiments of the present application provides a semantic segmentation method, which may include:

[0007] Acquire a first image and a second image; wherein the first image is an image to be semantically segmented, and the second image is an image frame previous to the first image;

[0008] performing optical flow calculation on the first image and the second image to obtain first optical flow data;

[0009] fusing the first optical flow data with the first image to obtain an optical flow fused image;

[0010] Perform semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image.

[0011] In a specific implementation of the first aspect, performing optical flow calculation on the first image and the second image to obtain first optical flow data may include:

[0012] Inputting the first image and the second image into a preset optical flow model, and obtaining an intermediate output result of the optical flow model; wherein the intermediate output result is an optical flow calculation result before the final output result of the optical flow model;

[0013] The intermediate output result is used as the first optical flow data.

[0014] In a specific implementation of the first aspect, fusing the first optical flow data with the first image to obtain an optical flow fused image may include:

[0015] interpolating the first optical flow data to obtain an interpolated optical flow result consistent with the size of the first image;

[0016] The interpolated optical flow data is fused with the first image to obtain the optical flow fused image.

[0017] In a specific implementation of the first aspect, fusing the interpolated optical flow data with the first image to obtain the optical flow fused image may include:

[0018] The interpolated optical flow result and the first image are spliced ​​in the channel dimension to obtain the optical flow fused image.

[0019] In a specific implementation of the first aspect, before performing semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image, the method may further include:

[0020] Acquire a first sample image, a second sample image, and a semantic segmentation annotation corresponding to the first sample image; wherein the second sample image is a frame image preceding the first sample image;

[0021] performing optical flow calculation on the first sample image and the second sample image to obtain first optical flow sample data;

[0022] fusing the first optical flow sample data with the first sample image to obtain an optical flow fused sample image corresponding to the first sample image;

[0023] The preset deep learning model is trained according to the optical flow fusion sample image and the semantic segmentation annotation to obtain the semantic segmentation model.

[0024] In a specific implementation of the first aspect, training a preset deep learning model based on the optical flow fusion sample image and the semantic segmentation annotation to obtain the semantic segmentation model may include:

[0025] Inputting the optical flow fusion sample image into the deep learning model and obtaining the output result of the deep learning model;

[0026] Calculating a training loss value of the deep learning model based on the output result and the semantic segmentation annotation;

[0027] The model parameters of the deep learning model are adjusted according to the training loss value to obtain the semantic segmentation model.

[0028] In a specific implementation of the first aspect, calculating the training loss value of the deep learning model according to the output result and the semantic segmentation annotation may include:

[0029] Calculating the segmentation region intersection-over-union ratio between the output result and the semantic segmentation annotation;

[0030] The training loss value is calculated according to the intersection-over-union ratio of the segmented regions.

[0031] A second aspect of the embodiments of the present application provides a semantic segmentation apparatus, which may include:

[0032] An image acquisition module, configured to acquire a first image and a second image; wherein the first image is an image to be semantically segmented, and the second image is an image frame preceding the first image;

[0033] an optical flow calculation module, configured to perform optical flow calculation on the first image and the second image to obtain first optical flow data;

[0034] a data fusion module, configured to fuse the first optical flow data with the first image to obtain an optical flow fused image;

[0035] The semantic segmentation module is used to perform semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image.

[0036] In a specific implementation of the second aspect, the optical flow calculation module may include:

[0037] a result acquisition unit, configured to input the first image and the second image into a preset optical flow model and acquire an intermediate output result of the optical flow model; wherein the intermediate output result is an optical flow calculation result before a final output result of the optical flow model;

[0038] A data determining unit is configured to use the intermediate output result as the first optical flow data.

[0039] In a specific implementation of the second aspect, the data fusion module may include:

[0040] a data interpolation unit, configured to interpolate the first optical flow data to obtain an interpolated optical flow result consistent with the size of the first image;

[0041] A data fusion unit is used to fuse the interpolated optical flow data with the first image to obtain the optical flow fused image.

[0042] In a specific implementation of the second aspect, the data fusion unit may include:

[0043] The dimension stitching subunit is used to stitch the interpolated optical flow result and the first image in the channel dimension to obtain the optical flow fusion image.

[0044] In a specific implementation of the second aspect, the semantic segmentation apparatus may further include:

[0045] A data acquisition module, configured to acquire a first sample image, a second sample image, and a semantic segmentation annotation corresponding to the first sample image; wherein the second sample image is an image frame preceding the first sample image;

[0046] a data calculation module, configured to perform optical flow calculation on the first sample image and the second sample image to obtain first optical flow sample data;

[0047] an image fusion module, configured to fuse the first optical flow sample data with the first sample image to obtain an optical flow fusion sample image corresponding to the first sample image;

[0048] The model training module is used to train a preset deep learning model based on the optical flow fusion sample image and the semantic segmentation annotation to obtain the semantic segmentation model.

[0049] In a specific implementation of the second aspect, the model training module may include:

[0050] a result acquisition unit, configured to input the optical flow fusion sample image into the deep learning model and obtain an output result of the deep learning model;

[0051] A loss value calculation unit, configured to calculate a training loss value of the deep learning model based on the output result and the semantic segmentation annotation;

[0052] A parameter adjustment unit is used to adjust the model parameters of the deep learning model according to the training loss value to obtain the semantic segmentation model.

[0053] In a specific implementation of the second aspect, the loss value calculation unit may include:

[0054] A first calculation subunit is used to calculate the segmentation region intersection-over-union ratio between the output result and the semantic segmentation annotation;

[0055] The second calculation subunit is used to calculate the training loss value according to the intersection-over-union ratio of the segmented regions.

[0056] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned semantic segmentation methods are implemented.

[0057] A fourth aspect of an embodiment of the present application provides a robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned semantic segmentation methods when executing the computer program.

[0058] A fifth aspect of the embodiments of the present application provides a computer program product, which, when run on a robot, enables the robot to perform the steps of any one of the above-mentioned semantic segmentation methods. Beneficial effects

[0059] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a first image and a second image; wherein the first image is an image to be semantically segmented, and the second image is an image of the previous frame of the first image; optical flow calculation is performed on the first image and the second image to obtain first optical flow data; the first optical flow data is fused with the first image to obtain an optical flow fused image; semantic segmentation is performed on the optical flow fused image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image. Through the embodiments of the present application, semantic segmentation can be performed in combination with the optical flow data of the image, thereby being able to mine the deep features of the image, better semantically segment unknown objects, and thereby effectively improving the accuracy of the semantic segmentation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0061] Figure 1 is a schematic flow chart of the semantic segmentation model training process;

[0062] Figure 2 is a schematic diagram of the training process of the optical flow model and the semantic segmentation model;

[0063] FIG3 is a flow chart of an embodiment of a semantic segmentation method according to an embodiment of the present application;

[0064] FIG4 is a structural diagram of an embodiment of a semantic segmentation device according to an embodiment of the present application;

[0065] FIG5 is a schematic block diagram of a robot in an embodiment of the present application. Modes for Carrying Out the Invention

[0066] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0067] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0068] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0069] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0070] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0071] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0072] With the development of intelligent technology, semantic segmentation technology has been applied to various scenarios such as autonomous driving, indoor navigation, virtual reality and image recognition.

[0073] In existing technologies, before using a semantic segmentation model to segment an image, it is usually necessary to train the model using a semantic segmentation training set. However, when the scene changes, unknown objects that are not included in the semantic segmentation training set may appear in the new scene, resulting in low accuracy of the semantic segmentation method.

[0074] In view of this, embodiments of the present application provide a semantic segmentation method, apparatus, computer-readable storage medium, and robot to address the low accuracy of existing semantic segmentation methods when faced with new scenarios. Through embodiments of the present application, semantic segmentation can be performed in conjunction with optical flow data of an image, thereby mining deep features of the image and better performing semantic segmentation on unknown objects, thereby effectively improving the accuracy of the semantic segmentation method.

[0075] It should be noted that the executor of the method of the present application is a robot, which may include but is not limited to any common robot in the prior art, such as a sweeping robot, an inspection robot, a food delivery robot, and a guiding robot.

[0076] In the embodiment of the present application, the optical flow calculation can be performed on the image using a preset optical flow model, and the semantic segmentation can be performed on the image using a preset semantic segmentation model. Therefore, the optical flow model and the semantic segmentation model in the embodiment of the present application will be first introduced below.

[0077] The optical flow model in the embodiment of the present application can be preferably a lightweight deep learning model. Through the optical flow model, optical flow estimation can be performed on the image. Specifically, the optical flow model may include an encoder (Encoder) and a decoder (Decoder), wherein the encoder can be mainly used for pyramid multi-scale feature extraction, and the decoder can be mainly used for optical flow estimation.

[0078] It is understandable that before using the optical flow model to perform optical flow estimation, the original deep learning model can be trained to obtain the optical flow model.

[0079] Specifically, two consecutive frames of sample images can be obtained, and the optical flow of the sample images can be annotated using any optical flow estimation method in the existing technology to obtain expected optical flow data (Ground Truth); then, the two frames of sample images can be used as the input of the original deep learning model, and the corresponding expected optical flow data can be used as the expected output of the original deep learning model to train the original deep learning model.

[0080] It can be understood that by traversing a preset number of two consecutive frames of sample images and repeating the above training process, the optical flow model of the embodiment of the present application can be obtained.

[0081] In a specific implementation of an embodiment of the present application, in order to ensure the training effect of the original deep learning model, the output predicted optical flow data can also be supervised by the expected optical flow data. Specifically, the model training loss value can be calculated based on the predicted optical flow data and the expected optical flow data output during the model training process. The calculation formula can be:

[0082] Wherein, x is the difference between the predicted optical flow data and the expected optical flow data, α and c are hyperparameters, and the values ​​of the two hyperparameters can be set according to actual needs, which is not specifically limited in the embodiment of the present application.

[0083] It can be understood that when the training loss value is less than the preset loss value threshold, or when the number of iterations of model training is greater than the preset number threshold, it can be considered that the obtained optical flow model is relatively accurate, thereby determining that the model training is completed.

[0084] In an embodiment of the present application, a semantic segmentation model can also be used to perform semantic segmentation on the image, wherein the semantic segmentation model can be any deep learning model in the prior art that can be used for semantic segmentation.

[0085] It is understood that before using the semantic segmentation model to perform semantic segmentation on an image, the original deep learning model can be trained to obtain a semantic segmentation model. Specifically, a semantic segmentation training sample set can be constructed, and the deep learning model can be trained using the semantic segmentation training sample set. Here, please refer to Figure 1. The training process of the semantic segmentation model may include the following steps:

[0086] Step S101: Obtain a first sample image, a second sample image, and a semantic segmentation annotation corresponding to the first sample image.

[0087] In an embodiment of the present application, a preset camera acquisition device can be used to capture video, and two adjacent frames of images in the video can be obtained. Here, the t-th frame image and the t-1-th frame image in the video can be obtained, where t is a positive integer, and the t-th frame image is recorded as the first sample image, and the t-1-th frame image is recorded as the second sample image.

[0088] In addition, the semantic segmentation annotation corresponding to the first sample image can also be obtained, wherein the semantic segmentation annotation of the image may include the number of pixels and pixel positions in the semantic segmentation area, that is, the number of pixels and pixel positions of each object in the image. Specifically, any semantic segmentation method in the prior art can be used to perform semantic segmentation to segment out each object in the first sample image and the area corresponding to each object.

[0089] Step S102: perform optical flow calculation on the first sample image and the second sample image to obtain first optical flow sample data.

[0090] It can be understood that the optical flow data of an image is a deep feature of the image and can be used to describe the movement of objects in the image, making it easier to distinguish between static objects and dynamic objects in the image. Based on this, the foreground and background of the image can be effectively distinguished. Therefore, when there are unknown objects in the image, introducing the optical flow data of the image helps to perform semantic segmentation of the unknown objects.

[0091] In an embodiment of the present application, an optical flow model can be used to calculate the optical flow of the first sample image and the second sample image. Specifically, referring to FIG2 , the first sample image and the second sample image can be input into the optical flow model. After that, the first sample image and the second sample image will pass through the encoder and the decoder. Here, the intermediate output result (Half Size output result) of the decoder can be used as the first optical flow sample data, wherein the intermediate output result is the output result before the final output result of the optical flow model. Specifically, the output result of the intermediate layer of the decoder can be used as the intermediate output result. Here, the size of the intermediate output result is half of the final output result (Half Size).

[0092] Step S103: Fusing the first optical flow sample data with the first sample image to obtain an optical flow fused sample image corresponding to the first sample image.

[0093] In an embodiment of the present application, the first optical flow sample data can be interpolated to obtain an interpolated sample optical flow result consistent with the size of the first sample image. Specifically, the first optical flow data can be interpolated using any interpolation method in the prior art, including but not limited to any common interpolation method in the prior art such as bilinear interpolation, bicubic interpolation and nearest neighbor interpolation. Preferably, bilinear interpolation can be used here to interpolate the first optical flow data.

[0094] Afterwards, the interpolated sample optical flow result and the first sample image can be concatenated in the channel dimension to obtain the optical flow fusion sample image, as shown in the following formula: Semantic =Concat(Upsample(Flow),Frame t )

[0095] Among them, Input Semantic is the optical flow fusion sample image, Flow is the first optical flow sample data, Upsample(Flow) is the interpolated sample optical flow result, Frame t is the first sample image, and Concat is the concatenation operation.

[0096] Step S104: Train the preset deep learning model based on the optical flow fusion sample image and semantic segmentation annotation to obtain a semantic segmentation model.

[0097] In an embodiment of the present application, the optical flow fusion sample image can be used as the input of the original deep learning model, and the semantic segmentation annotation corresponding to the optical flow fusion sample image can be used as the output of the original deep learning model to train the original deep learning model.

[0098] It is understandable that the structure of the original deep learning model can be specific and contextualized according to actual needs, and this application does not limit this. In the embodiments of this application, the original deep learning model preferably includes convolutional layers and pooling layers, and can perform common operations such as pyramid multi-scale feature extraction, pooling, and splicing.

[0099] In a specific implementation of the embodiment of the present application, the training loss value can also be calculated based on the output result of the original deep learning model and the semantic segmentation annotation. Here, the intersection-over-union (IoU) of the segmented area between the output result and the semantic segmentation annotation can be calculated, where the IoU can be used to represent the degree of overlap between the segmented areas of the two results. The calculation formula is as follows:

[0100] Specifically, dice is the intersection-over-union ratio of the segmented areas between the output result and the semantic segmentation annotation, X is the output result of the deep learning model, that is, the number of pixels and pixel positions of the output semantic segmentation area, Y is the semantic segmentation annotation, that is, the number of pixels and pixel positions of the semantic segmentation area of ​​the first sample image, |X∩Y| is the intersection between the output semantic segmentation area and the semantic segmentation area of ​​the first sample image, and |X|+|Y| is the union between the output semantic segmentation area and the semantic segmentation area of ​​the first sample image.

[0101] Afterwards, the training loss value L can be calculated based on the intersection-union ratio of the segmented area dice , the calculation formula can be as follows;

[0102] After obtaining the training loss value, the model parameters of the deep learning model can be adjusted according to the training loss value. For example, the parameters of the deep learning model can be adjusted using a reverse update method.

[0103] It is understandable that when the training loss value is less than the preset loss value threshold, or when the number of model training iterations is greater than the preset number threshold, the obtained semantic segmentation model can be considered to be relatively accurate. At this time, the semantic segmentation model can be used to perform semantic segmentation on the image. Referring to FIG3 , an embodiment of a semantic segmentation method in an embodiment of the present application may include:

[0104] Step S301: Acquire a first image and a second image.

[0105] In an embodiment of the present application, a monocular camera may be preferably used to capture video of the area to be semantically segmented, and the t-th frame image in the captured video may be used as the first image, and the t-1-th frame image may be used as the second image.

[0106] For example, the method of the present application can be specifically applied to the navigation process of a robot, and the area to be semantically segmented can be the driving area of ​​the robot. At this time, a monocular camera installed on the robot can be used to capture video of the driving area, and the t-th frame image in the captured video can be used as the first image, and the t-1-th frame image in the video can be used as the second image.

[0107] Step S302: perform optical flow calculation on the first image and the second image to obtain first optical flow data.

[0108] In an embodiment of the present application, the first image and the second image can be input into the trained optical flow model, and an intermediate output result of the optical flow model can be obtained, and the intermediate output result can be used as the first optical flow data.

[0109] Step S303: Fuse the first optical flow data with the first image to obtain an optical flow fused image.

[0110] In an embodiment of the present application, in order to enable the first optical flow data to be smoothly fused with the first image, the first optical flow data can be interpolated to obtain an interpolated optical flow result consistent with the size of the first image. Thereafter, the interpolated optical flow data and the first image can be spliced ​​in the channel dimension to obtain an optical flow fused image.

[0111] It is understandable that the specific content of step S303 can be referred to the relevant description of step S103, which will not be repeated here.

[0112] Step S304: Use a preset semantic segmentation model to perform semantic segmentation on the optical flow fusion image to obtain a semantic segmentation result corresponding to the first image.

[0113] In an embodiment of the present application, a trained semantic segmentation model can be used to perform semantic segmentation on the optical flow fusion image. Specifically, the optical flow fusion image can be used as input to the semantic segmentation model to obtain a semantic segmentation result corresponding to the first image. The semantic segmentation result may include information related to known objects and unknown objects.

[0114] In a specific implementation of the embodiment of the present application, the semantic segmentation result corresponding to the first image can be used for navigation, exploration, motion control and other related applications. For example, the method of the present application can be applied to a specific application scenario of navigation. The first image can be an image of a driving area. The semantic segmentation model can then be used to segment the driving area into objects such as drivable areas, obstacles, pedestrians, etc., and navigation can be performed accordingly.

[0115] In summary, the embodiment of the present application obtains a first image and a second image; wherein, the first image is an image to be semantically segmented, and the second image is an image of the previous frame of the first image; optical flow calculation is performed on the first image and the second image to obtain first optical flow data; the first optical flow data is fused with the first image to obtain an optical flow fused image; and the optical flow fused image is semantically segmented using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image. Through the embodiment of the present application, semantic segmentation can be performed in combination with the optical flow data of the image, thereby being able to mine the deep features of the image, better semantically segment unknown objects, and thereby effectively improving the accuracy of the semantic segmentation method.

[0116] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] Corresponding to the semantic segmentation method described in the above embodiment, FIG4 shows a structural diagram of an embodiment of a semantic segmentation device provided in an embodiment of the present application.

[0118] In this embodiment, a semantic segmentation device may include:

[0119] An image acquisition module 401 is configured to acquire a first image and a second image; wherein the first image is an image to be semantically segmented, and the second image is an image frame preceding the first image;

[0120] an optical flow calculation module 402, configured to perform optical flow calculation on the first image and the second image to obtain first optical flow data;

[0121] A data fusion module 403 is configured to fuse the first optical flow data with the first image to obtain an optical flow fused image;

[0122] The semantic segmentation module 404 is configured to perform semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image.

[0123] In a specific implementation of the embodiment of the present application, the optical flow calculation module may include:

[0124] a result acquisition unit, configured to input the first image and the second image into a preset optical flow model and acquire an intermediate output result of the optical flow model; wherein the intermediate output result is an optical flow calculation result before a final output result of the optical flow model;

[0125] A data determining unit is configured to use the intermediate output result as the first optical flow data.

[0126] In a specific implementation of the embodiment of the present application, the data fusion module may include:

[0127] a data interpolation unit, configured to interpolate the first optical flow data to obtain an interpolated optical flow result consistent with the size of the first image;

[0128] A data fusion unit is used to fuse the interpolated optical flow data with the first image to obtain the optical flow fused image.

[0129] In a specific implementation of the embodiment of the present application, the data fusion unit may include:

[0130] The dimension stitching subunit is used to stitch the interpolated optical flow result and the first image in the channel dimension to obtain the optical flow fusion image.

[0131] In a specific implementation of the embodiment of the present application, the semantic segmentation device may further include:

[0132] A data acquisition module, configured to acquire a first sample image, a second sample image, and a semantic segmentation annotation corresponding to the first sample image; wherein the second sample image is an image frame preceding the first sample image;

[0133] a data calculation module, configured to perform optical flow calculation on the first sample image and the second sample image to obtain first optical flow sample data;

[0134] an image fusion module, configured to fuse the first optical flow sample data with the first sample image to obtain an optical flow fusion sample image corresponding to the first sample image;

[0135] The model training module is used to train a preset deep learning model based on the optical flow fusion sample image and the semantic segmentation annotation to obtain the semantic segmentation model.

[0136] In a specific implementation of the embodiment of the present application, the model training module may include:

[0137] a result acquisition unit, configured to input the optical flow fusion sample image into the deep learning model and obtain an output result of the deep learning model;

[0138] A loss value calculation unit, configured to calculate a training loss value of the deep learning model based on the output result and the semantic segmentation annotation;

[0139] A parameter adjustment unit is used to adjust the model parameters of the deep learning model according to the training loss value to obtain the semantic segmentation model.

[0140] In a specific implementation of the embodiment of the present application, the loss value calculation unit may include:

[0141] A first calculation subunit is used to calculate the segmentation region intersection-over-union ratio between the output result and the semantic segmentation annotation;

[0142] The second calculation subunit is used to calculate the training loss value according to the intersection-over-union ratio of the segmented regions.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0145] Figure 5 shows a schematic block diagram of a robot provided in an embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0146] As shown in FIG5 , the robot 5 of this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, it implements the steps in the aforementioned semantic segmentation method embodiments, such as steps S301 to S304 shown in FIG3 . Alternatively, when the processor 50 executes the computer program 52, it implements the functions of the modules / units in the aforementioned apparatus embodiments, such as the functions of modules 401 to 404 shown in FIG4 .

[0147] For example, the computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 52 in the robot 5.

[0148] Those skilled in the art will understand that Figure 5 is merely an example of the robot 5 and does not constitute a limitation on the robot 5. The robot 5 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the robot 5 may also include input and output devices, network access devices, buses, etc.

[0149] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0150] The memory 51 can be an internal storage unit of the robot 5, such as a hard drive or memory of the robot 5. The memory 51 can also be an external storage device of the robot 5, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the robot 5. Furthermore, the memory 51 can include both an internal storage unit of the robot 5 and an external storage device. The memory 51 is used to store the computer program and other programs and data required by the robot 5. The memory 51 can also be used to temporarily store data that has been output or is about to be output.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0152] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0153] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] In the embodiments provided in this application, it should be understood that the disclosed devices / robots and methods can be implemented in other ways. For example, the device / robot embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0157] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0158] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A semantic segmentation method, characterized in that: include: Acquire a first image and a second image; wherein the first image is an image to be semantically segmented, and the second image is an image frame previous to the first image; Performing optical flow calculation on the first image and the second image to obtain first optical flow data; Fusing the first optical flow data with the first image to obtain an optical flow fused image; Perform semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image.

2. The semantic segmentation method according to claim 1, characterized in that: The performing optical flow calculation on the first image and the second image to obtain first optical flow data includes: Inputting the first image and the second image into a preset optical flow model, and obtaining an intermediate output result of the optical flow model; wherein the intermediate output result is an optical flow calculation result before a final output result of the optical flow model; The intermediate output result is used as the first optical flow data.

3. The semantic segmentation method according to claim 1, characterized in that: The fusing the first optical flow data with the first image to obtain an optical flow fused image includes: interpolating the first optical flow data to obtain an interpolated optical flow result consistent with the size of the first image; The interpolated optical flow data is fused with the first image to obtain the optical flow fused image.

4. The semantic segmentation method according to claim 3, characterized in that: The step of fusing the interpolated optical flow data with the first image to obtain the optical flow fused image includes: The interpolated optical flow result and the first image are spliced ​​in the channel dimension to obtain the optical flow fused image.

5. The semantic segmentation method according to any one of claims 1 to 4, characterized in that: Before performing semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image, the method further includes: Acquire a first sample image, a second sample image, and a semantic segmentation annotation corresponding to the first sample image; wherein the second sample image is a previous frame image of the first sample image; Performing optical flow calculation on the first sample image and the second sample image to obtain first optical flow sample data; Fusing the first optical flow sample data with the first sample image to obtain an optical flow fused sample image corresponding to the first sample image; The preset deep learning model is trained according to the optical flow fusion sample image and the semantic segmentation annotation to obtain the semantic segmentation model.

6. The semantic segmentation method according to claim 5, characterized in that: The step of training a preset deep learning model according to the optical flow fusion sample image and the semantic segmentation annotation to obtain the semantic segmentation model includes: Inputting the optical flow fusion sample image into the deep learning model, and obtaining the output result of the deep learning model; Calculating a training loss value of the deep learning model according to the output result and the semantic segmentation annotation; The model parameters of the deep learning model are adjusted according to the training loss value to obtain the semantic segmentation model.

7. The semantic segmentation method according to claim 6, characterized in that: The calculating the training loss value of the deep learning model according to the output result and the semantic segmentation annotation includes: Calculating the segmentation region intersection-over-union ratio between the output result and the semantic segmentation annotation; The training loss value is calculated according to the intersection-over-union ratio of the segmented regions.

8. A semantic segmentation device, characterized in that: include: An image acquisition module, used to acquire a first image and a second image; wherein the first image is an image to be semantically segmented, and the second image is an image frame previous to the first image; an optical flow calculation module, configured to perform optical flow calculation on the first image and the second image to obtain first optical flow data; A data fusion module, used for fusing the first optical flow data with the first image to obtain an optical flow fused image; The semantic segmentation module is used to perform semantic segmentation on the optical flow fusion image using a preset semantic segmentation model to obtain a semantic segmentation result corresponding to the first image.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the semantic segmentation method according to any one of claims 1 to 7 are implemented.

10. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the semantic segmentation method according to any one of claims 1 to 7 are implemented.

Images