Joint positioning information semantic segmentation method and device, equipment and storage medium

Abstract

The application provides a semantic segmentation method and device for joint positioning information, equipment and a storage medium, the method comprising: in the case that a sweeping robot performs a task, according to current positioning information of the sweeping robot, querying a cached encoder feature associated with the current positioning information; in the case that the cached encoder feature associated with the current positioning information is queried, comparing a current encoder feature with the cached encoder feature; in the case that the similarity between the current encoder feature and the cached encoder feature is greater than or equal to a preset similarity threshold, determining a cached semantic segmentation result associated with the current positioning information as a current semantic segmentation result. The method can improve the semantic segmentation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a semantic segmentation method and device combining positioning information, equipment and a storage medium. BACKGROUND

[0002] In the scenario of a sweeping robot, the perception and understanding of the scenario mainly rely on SLAM (Simultaneous Localization And Mapping, i.e., real-time positioning and map construction). SLAM enables the sweeping robot to plan a cleaning route, greatly improving the coverage and efficiency of cleaning. However, since SLAM cannot recognize the semantics of objects in space to distinguish the category information of the objects, the sweeping robot cannot well handle complex ground scenarios, is easily entangled by wires, is easily stuck by small objects such as socks and fruit shells, and lacks understanding of liquid stains in the cleaning process to spread the stains.

[0003] The understanding of scenario semantics is increasingly important for implementing a more intelligent sweeping robot. With the development of computer vision technology, combining semantic segmentation visual algorithms to enhance the perception and understanding of the scenario by the sweeping robot will gradually become an important direction for the intelligentization of sweeping robots.

[0004] Current semantic segmentation algorithms mainly use an encoder-decoder structure. The encoder performs feature extraction by down-sampling a high-resolution input image, and the decoder predicts the category of each pixel target of the input image according to the extracted features.

[0005] In the application scenario of a sweeping robot, the image resolution requirement is high, and the processing efficiency of scenario understanding and perception is sensitive. If the input image is directly subjected to semantic segmentation, it is difficult to meet the requirement for processing efficiency. SUMMARY

[0006] Therefore, the present application provides a semantic segmentation method and device combining positioning information, equipment and a machine-readable storage medium.

[0007] According to a first aspect of an embodiment of the present application, a semantic segmentation method combining positioning information is provided, comprising:

[0008] In the case that a sweeping robot performs a task, a cached encoder feature associated with the current positioning information of the sweeping robot is queried according to the current positioning information of the sweeping robot, wherein the current positioning information is the positioning information of the sweeping robot at the current position;

[0009] If a cached encoder feature associated with the current location information is found, the current encoder feature is compared with the cached encoder feature; wherein, the current encoder feature is obtained by reasoning on the current input image using the encoder of the semantic segmentation model, and the current input image is the image obtained by the sweeping robot at the current location;

[0010] If the similarity between the current encoder feature and the cache encoder feature is greater than or equal to a preset similarity threshold, the cache semantic segmentation result associated with the current positioning information is determined as the current semantic segmentation result.

[0011] According to a second aspect of the present application, a semantic segmentation device with joint positioning information is provided, which is deployed on a robotic vacuum cleaner. The device includes: a camera, a positioning module, and a processor; wherein: the camera is used to acquire images when the robotic vacuum cleaner is performing a task.

[0012] The positioning module is used to obtain the positioning information of the sweeping robot when the sweeping robot is performing a task;

[0013] The processor is configured to, when the robotic vacuum cleaner is performing a task, query the cached encoder features associated with the current positioning information of the robotic vacuum cleaner; wherein the current positioning information is the positioning information of the robotic vacuum cleaner at its current location; if the cached encoder features associated with the current positioning information are found, compare the current encoder features with the cached encoder features; wherein the current encoder features are obtained by reasoning on the current input image using the encoder of the semantic segmentation model, and the current input image is the image obtained by the robotic vacuum cleaner at its current location; if the similarity between the current encoder features and the cached encoder features is greater than or equal to a preset similarity threshold, determine the cached semantic segmentation result associated with the current positioning information as the current semantic segmentation result.

[0014] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first or second aspect.

[0015] According to a fourth aspect of the embodiments of this application, a storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0016] The semantic segmentation method of joint positioning information provided in the embodiments of the present application stores the encoder features, semantic segmentation results and positioning information in association, in the case that the sweeping robot performs a task, according to the current positioning information of the sweeping robot, the cached encoder features associated with the current positioning information are queried, in the case that the cached encoder features associated with the current positioning information are queried, the current encoder features are compared with the cached encoder features, and in the case that the similarity of the current encoder features and the cached encoder features is greater than or equal to a preset similarity threshold, the cached semantic segmentation results associated with the current positioning information are determined as the current semantic segmentation results, the spatio-temporal similarity of the application scenario is utilized to avoid a large amount of repeated calculation, and the semantic segmentation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flow diagram of a semantic segmentation method of joint positioning information provided by the embodiments of the present application;

[0018] Figure 2 is a working scene diagram of a sweeping robot provided by the embodiments of the present application;

[0019] Figure 3 is a semantic segmentation scene understanding diagram of a sweeping robot provided by the embodiments of the present application;

[0020] Figure 4 is an implementation flow diagram of a semantic segmentation method of joint positioning information provided by the embodiments of the present application;

[0021] Figure 5 is a flow diagram of a semantic segmentation method of joint positioning information provided by the embodiments of the present application;

[0022] Figure 6 is a structural diagram of a semantic segmentation device of joint positioning information provided by the embodiments of the present application;

[0023] Figure 7 is a hardware structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to designate the same elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of devices and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0025] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0026] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, characteristics and advantages of the embodiments of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the drawings.

[0027] Please refer to Figure 1 , a flowchart of a semantic segmentation method provided by the embodiments of the present application is shown, which can include the following steps: Figure 1

[0028] It should be noted that the sequence numbers of the steps in the embodiments of the present application do not mean the order of execution, the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] Step S100, in the case that the robot cleaner executes a task, according to the current positioning information of the robot cleaner, the cached encoder features associated with the current positioning information are queried; wherein the current positioning information is the positioning information of the robot cleaner at the current position.

[0030] In the embodiments of the present application, considering that in the application environment of the robot cleaner, there are a large number of areas that change little over time, such as the vicinity of fixed furniture items such as cabinets, televisions, refrigerators, and cleaned ground areas. For such areas, the spatiotemporal similarity of the application scene can be used to avoid a large amount of repeated calculation by caching the encoder features and the semantic segmentation results, thereby improving the efficiency of semantic segmentation.

[0031] For example, taking the common indoor working scene as shown in Figure 2 , the frequency of change of the fixed targets such as televisions, beds, and wardrobes and the cleaned ground area is low under the horizontal angle, and the semantic segmentation result obtained by performing semantic segmentation on these low-frequency change areas within a relatively short time interval is almost unchanged. If high-resolution semantic segmentation is performed on the low-frequency change areas every time within the unchanged time interval, a large amount of repeated calculation will be generated, resulting in low efficiency of the robot cleaner in understanding the scene.

[0032] In order to avoid a large amount of repeated calculation of similar scenes and improve the efficiency of high-resolution semantic segmentation of the robot cleaner, the spatiotemporal similarity of the working scene of the robot cleaner can be utilized.

[0033] ​Correspondingly, in a case where the sweeping robot performs a task, the sweeping robot can be positioned in real time, the positioning information of the sweeping robot is determined, and the cached encoder feature associated with the current positioning information of the sweeping robot is queried according to the current positioning information of the sweeping robot.

[0034] For example, the positioning information can be represented by a position (which can be determined by an xy coordinate) and an attitude angle (i.e., an orientation angle of the sweeping robot).

[0035] In a case where the cached encoder feature associated with the current positioning information is queried, the current encoder feature is compared with the cached encoder feature, wherein the current encoder feature is obtained by inferring a current input image by using an encoder of the semantic segmentation model, and the current input image is an image acquired by the sweeping robot at a current position.

[0036] In the embodiment of the present application, for the image acquired by the sweeping robot, the input image is no longer directly subjected to semantic segmentation by using the semantic segmentation model to determine a semantic segmentation result, but is subjected to semantic segmentation by using an encoder-decoder staged inference manner, that is, for the image acquired by the sweeping robot, the encoder of the semantic segmentation model is first used for inference to obtain an encoder feature, and in a case where it is determined that the decoder of the semantic segmentation model needs to be further used for inference, the decoder of the semantic segmentation model is then used for semantic segmentation.

[0037] Correspondingly, in a case where the cached encoder feature associated with the current positioning information is queried, the current encoder feature can be compared with the cached encoder feature queried.

[0038] For example, the current encoder feature is obtained by inferring a current input image by using an encoder of the semantic segmentation model, and the current input image is an image acquired by the sweeping robot at a current position.

[0039] It should be noted that in the embodiments of the present application, the current encoder feature can be determined according to the image obtained by the sweeping robot at the current position, and then the cached encoder feature associated with the current positioning information is queried, and then in the case that the cached encoder feature associated with the current positioning information is queried, the current encoder feature is compared with the cached encoder feature; or, the cached encoder feature associated with the current positioning information can be queried first, and in the case that the cached encoder feature associated with the current positioning information is queried, the current encoder feature is determined according to the image obtained by the sweeping robot at the current position, and the current encoder feature is compared with the cached encoder feature; or, the determination of the current encoder feature and the query of the cached encoder feature can be performed concurrently, and in the case that the cached encoder feature associated with the current positioning information is queried, the current encoder feature is compared with the cached encoder feature, and the specific implementation of the embodiments of the present application is not limited.

[0040] In step S120, in the case that the similarity between the current encoder feature and the cached encoder feature is greater than or equal to the preset similarity threshold, the cached semantic segmentation result associated with the current positioning information is determined as the current semantic segmentation result.

[0041] In the embodiments of the present application, in the case that the similarity between the cached encoder feature and the current encoder feature is greater than or equal to the preset similarity threshold, for example, the cosine similarity between the cached encoder feature and the current encoder feature is greater than or equal to the preset similarity threshold, or the L2 distance between the cached encoder feature and the current encoder feature is less than or equal to the preset distance threshold, the cached semantic segmentation result associated with the current positioning information can be determined as the current semantic segmentation result, without the need to use the decoder of the semantic segmentation model to infer the current encoder feature.

[0042] It can be seen that, in the method flow shown in Figure 1 In the method flow shown in the method flow shown in

[0043] In some embodiments, the above querying, according to the current positioning information of the sweeping robot, the cached encoder feature associated with the current positioning information can include:

[0044] determining, according to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region as the cached encoder feature associated with the current positioning information; wherein the target region is a region in the pre-divided regions that matches the current positioning information of the sweeping robot, and the regions are obtained by dividing the working environment of the sweeping robot according to the positioning information, and the positioning information is represented by a position and an attitude angle;

[0045] determining, according to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region as the cached encoder feature associated with the current positioning information; wherein the target region is a region in the pre-divided regions that matches the current positioning information of the sweeping robot, and the regions are obtained by dividing the working environment of the sweeping robot according to the positioning information, and the positioning information is represented by a position and an attitude angle;

[0046] For example, in order to reduce the amount of cached data and improve the query efficiency of cached data, the working environment of the sweeping robot can be divided into multiple regions according to the positioning information of the sweeping robot.

[0047] For example, the working environment of the sweeping robot can be divided into a grid, and for any grid, the grid center is taken as the positioning position, and 4 orientations (up, down, left, right) or 8 orientations (up, down, left, right, upper left, upper right, lower left, lower right) correspond to 4 or 8 different regions, respectively.

[0048] For example, according to the current position information of the sweeping robot, a region (referred to herein as a target region) that matches the current positioning information of the sweeping robot can be determined in the pre-divided regions, and the cached encoder feature associated with the target region can be determined as the cached encoder feature associated with the current positioning information.

[0049] In one example, the above determining, according to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region as the cached encoder feature associated with the current positioning information can include:

[0050] querying a global encoder feature modeling according to the current positioning information of the sweeping robot; wherein the global encoder feature modeling is used to cache the encoder features of the pre-divided regions;

[0051] determining, according to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region as the cached encoder feature associated with the current positioning information; wherein the target region is a region in the pre-divided regions that matches the current positioning information of the sweeping robot, and the regions are obtained by dividing the working environment of the sweeping robot according to the positioning information, and the positioning information is represented by a position and an attitude angle;

[0052] For example, the global encoder feature modeling can be used to cache the encoder features of the regions in the working environment of the sweeping robot.

[0053] Correspondingly, in a case where the robot cleaner performs a task, the global encoder feature modeling can be queried according to current positioning information of the robot cleaner, and the cached encoder feature associated with a target region in the global encoder feature modeling can be determined as the cached encoder feature associated with the current positioning information.

[0054] In some embodiments, the semantic segmentation method provided by the embodiments of the present application can further include:

[0055] In a case where the similarity between the current encoder feature and the cached encoder feature is less than a preset similarity threshold, the cached encoder feature associated with the current positioning information can be updated according to the current encoder feature; and

[0056] The current encoder feature can be inferred by using a decoder of the semantic segmentation model to determine a current semantic segmentation result, and the cached semantic segmentation result associated with the current positioning information can be updated according to the current semantic segmentation result.

[0057] For example, in a case where the cached encoder feature associated with the current positioning information is queried, but the similarity between the current encoder feature and the cached encoder feature is less than a preset similarity threshold, on the one hand, the cached encoder feature associated with the current positioning information can be updated according to the current encoder feature, for example, the cached encoder feature associated with the current positioning information is updated from the cached encoder feature to the current encoder feature.

[0058] On the other hand, the current encoder feature can be inferred by using a decoder of the semantic segmentation model to determine a current semantic segmentation result, and the cached semantic segmentation result associated with the current positioning information can be updated according to the current semantic segmentation result, for example, the cached semantic segmentation result associated with the current positioning information is updated from the cached semantic segmentation result to the current semantic segmentation result.

[0059] In some embodiments, the semantic segmentation method provided by the embodiments of the present application can further include:

[0060] In a case where the cached encoder feature associated with the current positioning information is not queried according to the current positioning information of the robot cleaner, for example, the robot cleaner moves to the region corresponding to the positioning information for the first time, on the one hand, the current encoder feature can be cached.

[0061] The current encoder feature can be inferred by using a decoder of the semantic segmentation model to determine a current semantic segmentation result, and the current semantic segmentation result can be cached.

[0062] For example, in a case where the cached encoder feature associated with the current positioning information is not queried according to the current positioning information of the robot cleaner, for example, the robot cleaner moves to the region corresponding to the positioning information for the first time, on the one hand, the current encoder feature can be cached.

[0063] On the other hand, the current encoder feature can be inferred by using the decoder of the semantic segmentation model, the current semantic segmentation result is determined, and the current semantic segmentation result is cached.

[0064] In the embodiment of the application, in the case where the semantic segmentation result corresponding to the current positioning information of the sweeping robot is determined, the downstream tasks such as obstacle avoidance and scene understanding can be performed according to the semantic segmentation result.

[0065] In order for those skilled in the art to better understand the technical solutions provided by the embodiments of the application, the technical solutions provided by the embodiments of the application will be described below in combination with specific application scenarios.

[0066] In this embodiment, considering the working scene of the sweeping robot, the common indoor working scene as shown in Figure 2 For example, in the case of the common indoor working scene as shown in FIG. 1, the fixed targets such as the television, the bed and the wardrobe and the low-frequency changing ground area change slowly in the time interval, and the semantic segmentation result obtained by performing semantic segmentation on these low-frequency changing areas is almost unchanged. If high-resolution semantic segmentation is performed on the low-frequency changing areas every time in the unchanged time interval, a large amount of repeated calculation will be generated, and the efficiency of the scene semantic understanding of the sweeping robot will become low.

[0067] In order to avoid a large amount of repeated calculation of similar scenes and improve the high-resolution semantic segmentation efficiency of the sweeping robot, the spatio-temporal similarity of the working scene of the sweeping robot can be utilized.

[0068] As shown in FIG. 2, by dividing the working environment of the sweeping robot into a plurality of coordinate areas according to the positioning information through the position and the attitude angle, for the area as shown in (a) of FIG. 3, the encoder feature of the semantic segmentation model can be cached as shown in (b) of FIG. 3, and a global encoder feature modeling can be established. Figure 2 Figure 3 As shown in (c) of FIG. 3, the semantic segmentation result output by the decoder can be cached, and a global encoder feature modeling can be established. Figure 3 Figure 3 As shown in (d) of FIG. 3, the scene understanding visualization schematic diagram can be shown. Figure 3

[0069] When the sweeping robot reaches the area again, the positioning offset is calibrated on the global encoder feature modeling to obtain the current cached encoder feature, and the difference between the cached encoder feature and the current encoder feature of the area is compared. For the area with small change difference, the cached semantic segmentation result is used to avoid repeated calculation and accelerate the semantic segmentation efficiency.

[0070] ​​​In order to improve the rationality of the matching of the positioning information, for the region divided in advance, when the region matching is performed according to the current positioning information, a certain error can be allowed to exist between the current positioning information and the positioning information of the matched region (such as the target region described above). For example, there is a tolerable error between the positions, and / or there is a tolerable error between the attitude angles, that is, the positioning offset calibration is realized.

[0071] Generally, the inference process of the encoder and the decoder of the semantic segmentation model is as shown in formula (1), the encoder and the decoder inference are inferred as a unified process, and in the highly similar scene of the sweeping robot application scenario, the decoder part will produce a large amount of repeated calculation.

[0072] Mask=Decoder(Encoder(x)) (1)

[0073] Wherein, x is an input image, Encoder is an encoder, Decoder is a decoder, and Mask is a semantic segmentation result.

[0074] In order to reduce the repeated calculation of the decoder in the similar scene and make full use of the semantic features of the input information, in this embodiment, a scheme of stage-by-stage inference of the encoder and the decoder is proposed, and the specific inference is shown in formula (2)-(4).

[0075] Feat=Encoder(x) (2)

[0076] S=F(Feat,Feat') (3)

[0077]

[0078] As shown in formula (3), the similarity S between the current encoder feature Feat and the previously cached encoder feature Feat' of the coordinate region is calculated by the similarity calculation function F; wherein S can be represented by cosine similarity or L2 distance.

[0079] It should be noted that in the case where S is represented by L2 distance, the larger the L2 distance, the smaller S; the smaller the L2 distance, the larger S.

[0080] As shown in formula (4), if the similarity S is greater than or equal to a set threshold T, the previously cached semantic segmentation result Mask' of the coordinate region is taken as the output result, so as to avoid the repeated calculation of the decoder in the similar scene; otherwise, the inference of the decoder is performed, and the cached encoder feature and the cached semantic segmentation result corresponding to the region are updated, so as to provide information for downstream tasks such as obstacle avoidance and scene modeling, and the main process diagram can be as shown in Figure 4 The implementation process can be as shown in Figure 5 .

[0081] The method provided by the application is described above. The device provided by the application is described below:

[0082] Please refer to Figure 6 A structure diagram of a joint positioning information semantic segmentation device provided by an embodiment of the application, wherein the joint positioning information semantic segmentation device can be deployed on a sweeping robot, such as Figure 6 As shown in the figure, the joint positioning information semantic segmentation device can include a camera 610, a positioning module 620, and a processor 630; wherein:

[0083] The camera 610 is configured to perform image acquisition when the sweeping robot performs a task.

[0084] The positioning module 620 is configured to obtain positioning information of the sweeping robot when the sweeping robot performs a task.

[0085] The processor 630 is configured to query a cached encoder feature associated with current positioning information of the sweeping robot according to the current positioning information when the sweeping robot performs a task; wherein the current positioning information is positioning information of the sweeping robot at a current position; in the case of querying the cached encoder feature associated with the current positioning information, compare a current encoder feature with the cached encoder feature; wherein the current encoder feature is obtained by using an encoder of a semantic segmentation model to infer a current input image, and the current input image is an image obtained by the sweeping robot at the current position; in the case that the similarity between the current encoder feature and the cached encoder feature is greater than or equal to a preset similarity threshold, determine a cached semantic segmentation result associated with the current positioning information as a current semantic segmentation result.

[0086] In some embodiments, the processor 630 queries the cached encoder feature associated with the current positioning information of the sweeping robot according to the current positioning information, including:

[0087] According to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region is determined as the cached encoder feature associated with the current positioning information; wherein the target region is a region in a pre-divided region that matches the current positioning information of the sweeping robot, and the region is obtained by dividing a working environment of the sweeping robot according to the positioning information of the sweeping robot, and the positioning information is represented by a position and an attitude angle.

[0088] In some embodiments, the processor 630 determines, according to the current positioning information of the sweeping robot, the cached encoder feature associated with the target area as the cached encoder feature associated with the current positioning information, including:

[0089] According to the current positioning information of the sweeping robot, querying a global encoder feature modeling, wherein the global encoder feature modeling is used to cache the encoder features of the pre-divided areas;

[0090] The cached encoder feature associated with the target area in the global encoder feature modeling is determined as the cached encoder feature associated with the current positioning information.

[0091] In some embodiments, the processor 630 is further configured to update the cached encoder feature associated with the current positioning information according to the current encoder feature in a case where the similarity between the current encoder feature and the cached encoder feature is less than a preset similarity threshold; and,

[0092] Infer the current encoder feature by using a decoder of a semantic segmentation model to determine a current semantic segmentation result, and update the cached semantic segmentation result associated with the current positioning information according to the current semantic segmentation result.

[0093] In some embodiments, the processor 630 is further configured to cache the current encoder feature in a case where the cached encoder feature associated with the current positioning information is not queried; and infer the current encoder feature by using a decoder of a semantic segmentation model to determine a current semantic segmentation result, and cache the current semantic segmentation result.

[0094] Please refer to Figure 7 A hardware structure schematic diagram of an electronic device is provided for the embodiments of the present application. The electronic device can include a processor 701 and a memory 702 storing machine executable instructions. The processor 701 and the memory 702 can communicate via a system bus 703. And by reading and executing the machine executable instructions in the memory 702, the processor 701 can perform the semantic segmentation method of joint positioning information described above.

[0095] The memory 702 mentioned herein can be any electronic, magnetic, optical or other physical storage device, which can contain or store information such as executable instructions, data, etc. For example, the machine readable storage medium can be RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard drive), solid state disk, any type of storage disk (such as optical disk, dvd, etc.), or similar storage medium, or combination thereof.

[0096] In some embodiments, there is also provided a storage medium, such as the storage medium 702 in the computer system 700, having stored therein machine-executable instructions, which when executed by a processor, implement the semantic segmentation method of joint localization information described above. For example, the machine-readable storage medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk and an optical data storage device, etc. Figure 7

[0097] It should be noted that, in the present document, the relational terms such as first and second and the like only serve to differentiate one entity or action from another, and do not necessarily require or imply any actual relationship or order between or among such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more limitations, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0098] The above description is merely illustrative of the application, and not restrictive. Modifications, equivalent replacements, improvements, and the like made to the application without departing from the spirit and principle of the application shall be included in the scope of the application.​

Claims

1. A method for semantic segmentation of joint localization information, characterized in that, The method comprises the following steps: In the case that the sweeping robot performs a task, the cached encoder feature associated with the current positioning information of the sweeping robot is queried according to the current positioning information of the sweeping robot; wherein the current positioning information is the positioning information of the sweeping robot at the current position; In the case that the cached encoder feature associated with the current positioning information is queried, the current encoder feature is compared with the cached encoder feature; wherein the current encoder feature is obtained by using the encoder of the semantic segmentation model to infer the current input image, and the current input image is the image obtained by the sweeping robot at the current position; In the case that the similarity between the current encoder feature and the cached encoder feature is greater than or equal to the preset similarity threshold, the cached semantic segmentation result associated with the current positioning information is determined as the current semantic segmentation result.

2. The method of claim 1, wherein, The method comprises the following steps: According to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region is determined as the cached encoder feature associated with the current positioning information; wherein the target region is the region in the pre-divided region that matches the current positioning information of the sweeping robot, and the region is obtained by dividing the working environment of the sweeping robot according to the positioning information of the sweeping robot, and the positioning information is represented by the position and the attitude angle.

3. The method of claim 2, wherein, The method comprises the following steps: According to the current positioning information of the sweeping robot, the global encoder feature modeling is queried; wherein the global encoder feature modeling is used to cache the encoder features of the pre-divided regions; The cached encoder feature associated with the target region in the global encoder feature modeling is determined as the cached encoder feature associated with the current positioning information.

4. The method of claim 1, wherein, The method further comprises the following steps: In the case that the similarity between the current encoder feature and the cached encoder feature is less than the preset similarity threshold, the cached encoder feature associated with the current positioning information is updated according to the current encoder feature; and The current semantic segmentation result is determined by using the decoder of the semantic segmentation model to infer the current encoder feature, and the cached semantic segmentation result associated with the current positioning information is updated according to the current semantic segmentation result.

5. The method of claim 1, wherein, The method further comprises the following steps: In the case that the cached encoder feature associated with the current positioning information is not queried, the current encoder feature is cached; and The current semantic segmentation result is determined by using the decoder of the semantic segmentation model to infer the current encoder feature, and the current semantic segmentation result is cached. 6.A semantic segmentation apparatus for joint localization information, characterized in that, The device is deployed on a sweeping robot, and the device comprises a camera, a positioning module and a processor; wherein: The camera is used to collect images in the case that the sweeping robot performs a task; The processor is used to perform the following steps: In the case that the sweeping robot performs a task, the cached encoder feature associated with the current positioning information of the sweeping robot is queried according to the current positioning information of the sweeping robot; wherein the current positioning information is the positioning information of the sweeping robot at the current position; In the case that the cached encoder feature associated with the current positioning information is queried, the current encoder feature is compared with the cached encoder feature; wherein the current encoder feature is obtained by using the encoder of the semantic segmentation model to infer the current input image, and the current input image is the image obtained by the sweeping robot at the current position; In the case that the similarity between the current encoder feature and the cached encoder feature is greater than or equal to the preset similarity threshold, the cached semantic segmentation result associated with the current positioning information is determined as the current semantic segmentation result. The method comprises the following steps: According to the current positioning information of the sweeping robot, the cached encoder feature associated with the target region is determined as the cached encoder feature associated with the current positioning information; wherein the target region is the region in the pre-divided region that matches the current positioning information of the sweeping robot, and the region is obtained by dividing the working environment of the sweeping robot according to the positioning information of the sweeping robot, and the positioning information is represented by the position and the attitude angle. The method comprises the following steps: According to the current positioning information of the sweeping robot, the global encoder feature modeling is queried; wherein the global encoder feature modeling is used to cache the encoder features of the pre-divided regions; The cached encoder feature associated with the target region in the global encoder feature modeling is determined as the cached encoder feature associated with the current positioning information. The method further comprises the following steps: In the case that the similarity between the current encoder feature and the cached encoder feature is less than the preset similarity threshold, the cached encoder feature associated with the current positioning information is updated according to the current encoder feature; and The current semantic segmentation result is determined by using the decoder of the semantic segmentation model to infer the current encoder feature, and the cached semantic segmentation result associated with the current positioning information is updated according to the current semantic segmentation result. The method further comprises the following steps: In the case that the cached encoder feature associated with the current positioning information is not queried, the current encoder feature is cached; and The current semantic segmentation result is determined by using the decoder of the semantic segmentation model to infer the current encoder feature, and the current semantic segmentation result is cached. The positioning module is configured to acquire positioning information of the sweeping robot when the sweeping robot is performing a task. The processor is configured to, when the sweeping robot is performing a task, query a cached encoder feature associated with current positioning information of the sweeping robot according to the current positioning information, the current positioning information being positioning information of the sweeping robot at a current position; when a cached encoder feature associated with the current positioning information is queried, compare a current encoder feature with the cached encoder feature, the current encoder feature being obtained by inferring a current input image using an encoder of a semantic segmentation model, the current input image being an image acquired by the sweeping robot at the current position; and when a similarity between the current encoder feature and the cached encoder feature is greater than or equal to a preset similarity threshold, determine a cached semantic segmentation result associated with the current positioning information as a current semantic segmentation result.

7. The apparatus of claim 6, wherein, The processor queries a cached encoder feature associated with current positioning information of the sweeping robot according to the current positioning information, and includes: The processor determines, according to the current positioning information of the sweeping robot, a cached encoder feature associated with a target region as the cached encoder feature associated with the current positioning information, the target region being a region in pre-divided regions that matches the current positioning information of the sweeping robot, the regions being obtained by dividing a working environment of the sweeping robot according to positioning information of the sweeping robot, the positioning information being represented by a position and an attitude angle.

8. The apparatus of claim 7, wherein, The processor determines, according to the current positioning information of the sweeping robot, a cached encoder feature associated with a target region as the cached encoder feature associated with the current positioning information, and includes: The processor queries a global encoder feature modeling according to the current positioning information of the sweeping robot, the global encoder feature modeling being used to cache encoder features of the pre-divided regions; The processor determines, in the global encoder feature modeling, a cached encoder feature associated with the target region as the cached encoder feature associated with the current positioning information.

9. The apparatus of claim 6, wherein The processor is further configured to, when a similarity between the current encoder feature and the cached encoder feature is less than a preset similarity threshold, update the cached encoder feature associated with the current positioning information according to the current encoder feature. In addition, The processor is further configured to, when the current encoder feature is not queried, cache the current encoder feature; and infer the current encoder feature using a decoder of the semantic segmentation model to determine a current semantic segmentation result, and update a cached semantic segmentation result associated with the current positioning information according to the current semantic segmentation result.

10. The apparatus of claim 6, wherein The processor is further configured to, when the current encoder feature is not queried, cache the current encoder feature; and infer the current encoder feature using a decoder of the semantic segmentation model to determine a current semantic segmentation result, and cache the current semantic segmentation result. ​ 11. An electronic device, comprising: A computer program product comprising a processor and a memory having stored therein machine executable instructions executable by the processor to implement the method of any of claims 1-5.

12. A storage medium, characterized by A storage medium having stored therein machine executable instructions to implement the method of any of claims 1-5 when executed by a processor.

Images