Multi-scale road surface full-element semantic segmentation method and device, electronic equipment and storage medium

Abstract

This application discloses a multi-scale road surface full-element semantic segmentation method, apparatus, electronic device, and storage medium. The method includes: acquiring image information from a preset scene; obtaining segmentation results from the image information using a pre-trained semantic segmentation model, wherein the semantic segmentation model is trained using multiple sets of data through machine learning, and each set of data includes: image data with variable resolution and a resolution scaling factor corresponding to the image data; and obtaining segmentation results for each element at different resolutions in the preset scene based on the segmentation results. This application achieves multi-scale training and prediction, improving the segmentation effect of objects of different sizes in images of a preset scene.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a multi-scale road surface full-element semantic segmentation method, device, electronic device, and storage medium. Background Technology

[0002] Panoramic segmentation of street scenes refers to semantic segmentation of all semantic elements in an image and obtaining the segmentation results of the semantic elements.

[0003] Semantic segmentation networks typically train themselves using data at the same scale (resolution) during semantic segmentation. Higher resolution input data results in better segmentation of small objects, such as distant road signs, pedestrians, and traffic lights. However, high resolution makes it difficult for the network to capture global semantic information of larger objects, such as nearby ground, vehicles, and pedestrians, leading to poorer segmentation of large objects. Conversely, lower resolution input data makes it easier to capture global semantic information of large objects, resulting in better segmentation, but often overlooking smaller objects in the image.

[0004] Therefore, in general, appropriately increasing the resolution is not good for large targets but good for small targets, while appropriately decreasing the resolution is good for large targets but not good for small targets. Semantic segmentation networks in related technologies struggle to simultaneously achieve good recognition results for both large and small targets. Summary of the Invention

[0005] This application provides a multi-scale road surface full-element semantic segmentation method, device, electronic device, and storage medium to achieve multi-scale training and prediction, and improve the segmentation effect of objects of different sizes in images.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a multi-scale road surface full-element semantic segmentation method, wherein the method includes: acquiring image information in a preset scene; obtaining segmentation results in the image information through a pre-trained semantic segmentation model, wherein the semantic segmentation model is trained using multiple sets of data through machine learning, and each set of data includes: image data with variable resolution and a resolution scaling factor corresponding to the image data; during training, the image resolution of the same batch of image data is the same, the image resolution of different batches of image data is different, and the multiple sets of data after training have a preset resolution variation range; and obtaining the segmentation results of each element in the preset scene at different resolutions based on the segmentation results.

[0008] Secondly, embodiments of this application also provide a semantic segmentation apparatus, wherein the apparatus includes: an acquisition module for acquiring image information in a preset scene; a training module for obtaining segmentation results in the image information through a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by machine learning training using multiple sets of data, each set of data including: image data with variable resolution and a resolution scaling factor corresponding to the image data; during training, the image resolution of the same batch of image data is the same, the image resolution of different batches of image data is different, and the multiple sets of data after training have a preset resolution variation range; and a segmentation module for obtaining the segmentation results of each element in the preset scene at different resolutions based on the segmentation results.

[0009] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the above-described method.

[0010] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the above-described method.

[0011] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0012] After acquiring image information from a preset scene, a pre-trained semantic segmentation model is used to obtain segmentation results from the image information. Since each set of data includes image data with variable resolution and a resolution scaling factor corresponding to the image data, and the image resolution is the same for the same batch of image data during training, while the image resolution is different for different batches of image data, and there is a preset resolution variation range in the multiple sets of data after training, a multi-scale training and prediction method is provided to obtain the segmentation results of each element at different resolutions in the preset scene. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 This is a flowchart illustrating a multi-scale road surface full-element semantic segmentation method in an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of the structure of a semantic segmentation device according to an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] This application provides a method, such as... Figure 1 As shown, a flowchart of an embodiment of this application is provided, and the method includes at least the following steps S110 to S130:

[0020] Step S110: Obtain image information from the preset scene.

[0021] For street scenes that require semantic segmentation, image information from a preset scene can be obtained.

[0022] The methods for acquiring images may include, but are not limited to, acquisition by the vehicle, acquisition by the roadside, and synchronization with the cloud. No specific limitations are imposed in the embodiments of this application.

[0023] Step S120: Obtain the segmentation result in the image information through a pre-trained semantic segmentation model. The semantic segmentation model is trained by machine learning using multiple sets of data. Each set of data includes: image data with variable resolution and the resolution scaling factor corresponding to the image data.

[0024] In practice, during training, randomly generated scaling factors within a preset range are used to scale the images before they are input into the network for training.

[0025] Afterwards, when the trained model makes predictions, it needs to scale the images using several other pre-set scaling factors. Then, the scaled images are input into the model one by one, and the model outputs results at different resolutions. Finally, the results at different resolutions are fused to produce the final semantic segmentation result.

[0026] Preferably, during training, the image resolution of image data in the same batch is the same, the image resolution of image data in different batches is different, and the multiple sets of data after training have a preset range of resolution variation.

[0027] A pre-defined semantic segmentation model can be trained using multiple sets of data through machine learning. Each set of data includes: image data with variable resolution and a resolution scaling factor corresponding to the image data. However, there are certain requirements for the image data used in the training set; that is, the data resolution or the range of variation of the data resolution is set before training.

[0028] Furthermore, during training, image data of different resolutions are trained using the resolution scaling factor corresponding to the image data. To achieve better segmentation results, the image resolution is the same for image data within the same batch, while the image resolution is different for image data from different batches. Thus, training with the same image resolution for image data within the same batch yields the desired result, while training with different image resolutions for image data from different batches yields the corresponding results.

[0029] In addition, in order to ensure that the scaling resolution is within a suitable range, a preset resolution variation range is also required for the multiple sets of trained data.

[0030] Preferably, the machine learning is deep learning.

[0031] Step S130: Based on the segmentation results, obtain the segmentation results of each element in the preset scene at different resolutions.

[0032] By adding the semantic segmentation results (prediction results) of different groups (different resolutions), the segmentation result of each element at different resolutions in the preset scene is obtained.

[0033] The new feature network fusion structure can fully integrate features of different sizes, enabling the semantic network to acquire richer contextual information.

[0034] In one embodiment of this application, the segmentation result in the image information is obtained through a pre-trained semantic segmentation model. The semantic segmentation model is trained using multiple sets of data via machine learning. Each set of data includes: image data with variable resolution and a resolution scaling factor corresponding to the image data. During training, the image resolution is the same for image data in the same batch, and different resolutions for image data in different batches. Furthermore, the trained sets of data have a preset resolution variation range. This includes: multi-scale training of the semantic segmentation model's network structure based on the image data with variable resolution and the resolution scaling factor corresponding to the image data; and obtaining a multi-scale prediction result in the image information based on the preset resolution variation range in the multiple sets of data.

[0035] In practice, the semantic segmentation model's network structure is trained using multi-scale methods based on the variable-resolution image data and the corresponding resolution scaling factor. The scaling factor must be within a preset range. Then, based on the preset resolution variation range in the multiple sets of data, multi-scale prediction results in the image information are obtained. The resolution scaling factor can be an empirical value, or it can be different in different scenarios, and can be selected or configured according to the actual scenario.

[0036] In one embodiment of this application, the pre-trained semantic segmentation model further includes: setting the first resolution of the sample data input into the network structure to twice that of the original image and the second resolution to 0.5 times that of the original image during training, thereby obtaining a resolution scaling factor in the range of {0.5-2.0}.

[0037] In practice, during training, the first resolution of the sample data input into the network structure is set to twice that of the original image, and the second resolution is set to 0.5 times that of the original image, resulting in a resolution scaling factor in the range of {0.5-2.0}. Considering street scene scenarios, the scaling of the original image should not be too large.

[0038] In some embodiments, zoom cameras can be configured according to their focal length. Image distortion also needs to be considered.

[0039] In one embodiment of this application, the semantic segmentation model uses Bisenetv2 as the network model to be trained.

[0040] The detail branch in Bisenetv2 is responsible for spatial details. This branch requires rich channel capacity to encode rich spatial detail information. Wide channels and shallow layers are used to handle spatial details.

[0041] The semantic branch in Bisenetv2 is designed to capture high-level semantics. The ratio of the semantic branch to the detail branch is λ (λ < 1), making this branch relatively lightweight. The semantic branch can use any lightweight convolutional mode while employing a fast downsampling strategy to increase the level of feature representation and rapidly expand the receptive field. High-level semantics requires a larger receptive field; therefore, the semantic branch uses global average pooling to embed the global contextual response.

[0042] When performing semantic segmentation, the Bisenetv2 semantic segmentation model typically trains the network using data at the same scale (resolution). When the input data has a high resolution, it performs well on small objects, such as distant road signs, pedestrians, and traffic lights. However, high resolution makes it difficult for the network to capture global semantic information of large objects, such as nearby ground, vehicles, and pedestrians, thus worsening the segmentation of large objects. Conversely, when the input data has a low resolution, it is easier to capture global semantic information of large objects, resulting in better segmentation of large objects, but often overlooking small objects in the image.

[0043] In one embodiment of this application, obtaining the segmentation result in the image information through a pre-trained semantic segmentation model includes: using the pre-trained semantic segmentation model, pre-setting a first resolution scaling factor, a second resolution scaling factor, a third resolution scaling factor, and a fourth resolution scaling factor; scaling the image data to 0.5 times, 1.0 times, 1.5 times, and 2.0 times the original image size respectively based on the first resolution scaling factor, the second resolution scaling factor, the third resolution scaling factor, and the fourth resolution scaling factor to obtain four different prediction results; and adding the four different preset results to obtain the segmentation result in the image information.

[0044] In specific implementation, the first resolution scaling factor, the second resolution scaling factor, the third resolution scaling factor, and the fourth resolution scaling factor are preset through the pre-trained semantic segmentation model.

[0045] When using the trained model for prediction, several different scaling factors are pre-set: a first resolution scaling factor, a second resolution scaling factor, a third resolution scaling factor, and a fourth resolution scaling factor [0.5, 1.0, 1.5, 2.0]. The image data to be read is scaled to 0.5 times, 1.0 times, 1.5 times, and 2.0 times the original image size, respectively. Based on the first, second, third, and fourth resolution scaling factors, the image data is scaled to 0.5, 1.0, 1.5, and 2.0 times the original image size, respectively, resulting in four different prediction results. Then, the data generated using different scaling factors is input into the trained semantic segmentation model for prediction, producing four different prediction results. Finally, these four different prediction results are added together to obtain the final result.

[0046] In one embodiment of this application, obtaining the segmentation result in the image information through a pre-trained semantic segmentation model further includes: obtaining global semantic information in the current target image through the pre-trained semantic segmentation model, the current target image, and the resolution scaling factor corresponding to the current target image; and segmenting each element in the image information according to the global semantic information.

[0047] In practice, the global semantic information here is determined after processing with a preset resolution scaling factor, so it is compatible with both large and small targets. Each element in the image information can be segmented based on the global semantic information. When the resolution of the data input into the network is well-suited to the target's attributes, the global semantic information for both large and small targets can be obtained.

[0048] In one embodiment of this application, the variable resolution image data and the corresponding resolution scaling factor of the image data include: first image data with variable sample data resolution and a first resolution scaling factor corresponding to the first image data, wherein the first image data includes at least one of the following: distant road signs, distant pedestrians, traffic lights, and the first resolution scaling factor is not greater than 2.0; and second image data with variable sample data resolution and a second resolution scaling factor corresponding to the second image data, wherein the second image data includes at least one of the following: near ground, vehicles, near pedestrians, and the second resolution scaling factor is not less than 0.5.

[0049] For distant road signs, distant pedestrians, and traffic lights in the preset scene to be segmented, the first resolution scaling factor should not exceed 2.0. The distant scenes should be appropriately enlarged.

[0050] For the pre-defined scene to be segmented, including the near-end ground, vehicles, and near-end pedestrians, the second resolution scaling factor is no less than 0.5. In other words, the near-end scene is appropriately scaled down.

[0051] It is understood that the above-mentioned semantic segmentation device can realize all the steps of the multi-scale road surface full-element semantic segmentation method provided in the foregoing embodiments. The relevant explanations of the multi-scale road surface full-element semantic segmentation method are applicable to the semantic segmentation device, and will not be repeated here.

[0052] This application also provides a multi-scale training method for training a semantic segmentation model, wherein the method includes:

[0053] The semantic segmentation model is derived by training multiple sets of data using machine learning.

[0054] Preferably, the Bisenetv2 network is used as the main network structure for training.

[0055] Each set of data used in training includes: image data with variable resolution and the resolution scaling factor corresponding to the image data; during training, the image resolution of the same batch of image data is the same, the image resolution of different batches of image data is different, and the multiple sets of data after training have a preset resolution variation range.

[0056] In some embodiments, the network structure of the semantic segmentation model is trained using multi-scale methods based on the variable resolution image data and the resolution scaling factor corresponding to the image data; and multi-scale prediction results in the image information are obtained based on the preset resolution variation range in the multiple sets of data.

[0057] In some embodiments, during training, the first resolution of the sample data input into the network structure is set to twice that of the original image, and the second resolution is set to 0.5 times that of the original image, resulting in a resolution scaling factor in the range of {0.5-2.0}.

[0058] In some embodiments, a first resolution scaling factor, a second resolution scaling factor, a third resolution scaling factor, and a fourth resolution scaling factor are preset through the pre-trained semantic segmentation model;

[0059] Based on the first resolution scaling factor, the second resolution scaling factor, the third resolution scaling factor, and the fourth resolution scaling factor, the image data is scaled to 0.5 times, 1.0 times, 1.5 times, and 2.0 times the original image, respectively, resulting in four different prediction results;

[0060] The segmentation result in the image information is obtained by adding the four different preset results together.

[0061] In some embodiments, global semantic information in the current target image is obtained by using a pre-trained semantic segmentation model, the current target image, and the resolution scaling factor corresponding to the current target image.

[0062] Based on the global semantic information, each element in the image information is segmented.

[0063] In some embodiments, the sample data includes first image data with variable resolution and a first resolution scaling factor corresponding to the first image data, wherein the first image data includes at least one of the following: a distant road sign, a distant pedestrian, and a traffic light, and the first resolution scaling factor is not greater than 2.0.

[0064] The sample data includes second image data with variable resolution and a second resolution scaling factor corresponding to the second image data, wherein the second image data includes at least one of the following: near-end ground, vehicles, near-end pedestrians, and the second resolution scaling factor is not less than 0.5.

[0065] This application embodiment also provides a semantic segmentation device 200, such as Figure 2 The diagram shows a schematic representation of a semantic segmentation device in an embodiment of this application. The device 200 includes at least: an acquisition module 210, a training module 220, and a segmentation module 230, wherein:

[0066] In one embodiment of this application, the acquisition module 210 is specifically used to: acquire image information in a preset scene for a street scene that is to be semantically segmented.

[0067] The methods for acquiring images may include, but are not limited to, acquisition by the vehicle, acquisition by the roadside, and synchronization with the cloud. No specific limitation is made in the embodiments of this application.

[0068] In one embodiment of this application, the training module 220 is specifically used to: train a preset semantic segmentation model using multiple sets of data through machine learning, where each set of data includes: image data with variable resolution and a resolution scaling factor corresponding to the image data. The sample dataset contains image data, but there are certain requirements for this training image data; that is, the data resolution or the range of variation of the data resolution is set before training.

[0069] Furthermore, during training, image data of different resolutions are trained using the resolution scaling factor corresponding to the image data. To achieve better segmentation results, the image resolution is the same for image data within the same batch, while the image resolution is different for image data from different batches. Thus, training with the same image resolution for image data within the same batch yields the desired result, while training with different image resolutions for image data from different batches yields the corresponding results.

[0070] In addition, to ensure that the scaling resolution is within a suitable range, a preset resolution variation range is also required for the multiple sets of trained data.

[0071] In one embodiment of this application, the segmentation module 230 is specifically used to: obtain the segmentation result of each element at different resolutions in the preset scene by adding the semantic segmentation results (prediction results) of different groups (different resolutions).

[0072] The new feature network fusion structure can fully integrate features of different sizes, enabling the semantic network to acquire richer contextual information.

[0073] It is understood that the above-mentioned semantic segmentation device can realize all the steps of the multi-scale road surface full-element semantic segmentation method provided in the foregoing embodiments. The relevant explanations of the multi-scale road surface full-element semantic segmentation method are applicable to the semantic segmentation device, and will not be repeated here.

[0074] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0075] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0076] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0077] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a semantic segmentation device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0078] Acquire image information from a preset scene;

[0079] The segmentation result in the image information is obtained by using a pre-trained semantic segmentation model. The semantic segmentation model is trained by machine learning using multiple sets of data. Each set of data includes: image data with variable resolution and the resolution scaling factor corresponding to the image data. During training, the image resolution of the same batch of image data is the same, and the image resolution of different batches of image data is different. Moreover, the multiple sets of data after training have a preset resolution variation range.

[0080] Based on the segmentation results, the segmentation results of each element at different resolutions in the preset scene are obtained.

[0081] The above is as stated in this application. Figure 1The semantic segmentation apparatus disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0082] The electronic device can also perform Figure 1 The method for executing semantic segmentation devices, and the implementation of semantic segmentation devices in... Figure 1 The functions of the embodiments shown are not described in detail here.

[0083] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The semantic segmentation device in the illustrated embodiment executes a method, specifically for performing the following:

[0084] Acquire image information from a preset scene;

[0085] The segmentation result in the image information is obtained by using a pre-trained semantic segmentation model. The semantic segmentation model is trained by machine learning using multiple sets of data. Each set of data includes: image data with variable resolution and the resolution scaling factor corresponding to the image data. During training, the image resolution of the same batch of image data is the same, and the image resolution of different batches of image data is different. Moreover, the multiple sets of data after training have a preset resolution variation range.

[0086] Based on the segmentation results, the segmentation results of each element at different resolutions in the preset scene are obtained.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0092] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0093] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0094] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multi-scale pavement full-factor semantic segmentation method, wherein, The method comprises: acquiring image information in a preset scene; obtaining a segmentation result in the image information by using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by using multiple sets of data for machine learning training, and each set of data comprises image data with variable resolution and a resolution scaling factor corresponding to the image data; during training, the semantic segmentation model uses randomly generated scaling factors within a preset interval to scale the image and then inputs the scaled image into the network for training; during training, the image resolution of the same batch of image data is the same, the image resolution of different batches of image data is different, and the multiple sets of data after training have a preset resolution variation range; during prediction of the trained semantic segmentation model, the image is scaled using another preset scaling factor, and then the scaled image is sequentially input into the model, the model outputs results of different resolutions, and then the results of different resolutions are fused to obtain the final semantic segmentation result; according to the segmentation result, the segmentation result of each element in the preset scene with different resolutions is obtained.

2. The method of claim 1, wherein, obtaining a segmentation result in the image information by using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by using multiple sets of data for machine learning training, and each set of data comprises image data with variable resolution and a resolution scaling factor corresponding to the image data; comprising: according to the image data with variable resolution and the resolution scaling factor corresponding to the image data, the network structure of the semantic segmentation model is trained in multiple scales; based on the preset resolution variation range in the multiple sets of data, a multi-scale prediction result in the image information is obtained.

3. The method of claim 2, wherein, the pre-trained semantic segmentation model further comprises: during training, the first resolution of the sample data in the input network structure is set to 2 times the original picture, and the second resolution is set to 0.5 times the original picture, to obtain a resolution scaling factor in the interval {0.5-2.0}.

4. The method of claim 2, wherein, the semantic segmentation model selects Bisenetv2 as the network model to be trained.

5. The method of claim 1, wherein, obtaining a segmentation result in the image information by using a pre-trained semantic segmentation model, comprising: pre-setting a first resolution scaling factor, a second resolution scaling factor, a third resolution scaling factor and a fourth resolution scaling factor by using the pre-trained semantic segmentation model; scaling the image data to 0.5 times, 1.0 times, 1.5 times and 2.0 times the original picture based on the first resolution scaling factor, the second resolution scaling factor, the third resolution scaling factor and the fourth resolution scaling factor, to obtain four different prediction results; adding the four different preset results to obtain the segmentation result in the image information.

6. The method of claim 1, wherein, obtaining a segmentation result in the image information by using a pre-trained semantic segmentation model, further comprising: obtaining global semantic information in the current target image by a pre-trained semantic segmentation model and the current target image and a resolution scaling factor corresponding to the current target image; segmenting each element in the image information according to the global semantic information.

7. The method of claim 1, wherein, The resolution-variable image data and the resolution scaling factor corresponding to the image data include: sample data resolution-variable first image data and a first resolution scaling factor corresponding to the first image data, wherein the first image data at least includes one of the following: a remote sign, a remote pedestrian, a traffic light, and the first resolution scaling factor is not greater than 2.0; sample data resolution-variable second image data and a second resolution scaling factor corresponding to the second image data, wherein the second image data at least includes one of the following: a near ground, a vehicle, a near pedestrian, and the second resolution scaling factor is not less than 0.

5.

8. A multi-scale road surface full-factor semantic segmentation apparatus, wherein, The device includes: an acquisition module configured to acquire image information in a preset scene; a training module configured to obtain a segmentation result in the image information by a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by machine learning using a plurality of sets of data, and each set of data includes resolution-variable image data and a resolution scaling factor corresponding to the image data; the semantic segmentation model uses randomly generated scaling factors within a preset range to scale the image during training, and then inputs the scaled image into the network for training; the image resolution of the image data in the same batch is the same during training, the image resolution of the image data in different batches is different, and the plurality of sets of data after training have a preset resolution variation range; during prediction of the trained semantic segmentation model, the image is scaled using another preset scaling factor, then the scaled image is sequentially input into the model, the model outputs results of different resolutions, and then the results of different resolutions are fused to generate the final semantic segmentation result; a segmentation module configured to obtain a segmentation result of each element in the preset scene at different resolutions according to the segmentation result. 9.An electronic device, comprising: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the method of any of claims 1-7. 10.A computer readable storage medium storing one or more programs, which when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any of claims 1-7.

Images