Visual surface attribute classification prediction model training method and system

Abstract

The application provides a kind of visual ground attribute classification prediction model training method and system, belong to automatic driving technical field, obtain training data;Visual ground attribute classification prediction model is trained based on training data, wherein, visual ground attribute classification prediction model includes vision module, body perception module and classification network;The input of vision module is 1 frame image block, and the output is ground attribute label;The input of body perception module is 100 milliseconds of accelerometer and the time sequence signal of wheel speed encoder, and the output is ground attribute label;Classification network learns the model parameters of vision module and body perception module in a full-supervised manner based on artificial annotation samples, and obtains ground attribute classification.The application utilizes the stability of body perception to environmental changes and other characteristics, constructs supervision samples using the body perception results of the same ground, optimizes the vision perception model online, so that it has the self-adaptive ability to environmental changes, and improves the robustness.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a method and system for training a visual surface attribute classification and prediction model. Background Technology

[0002] In complex scenarios, environmental perception data from LiDAR, cameras, and other sensors can be used to classify and identify the surface attributes of the area in front of the unmanned platform, thereby predicting passability and the cost of passage and establishing safe driving strategies accordingly. Compared to LiDAR, visual images captured by cameras have rich texture features and are widely used. However, in complex scenarios, surface features are diverse and dynamically changing, and are affected by factors such as lighting, season, and weather. Vision-based methods lack sufficient robustness and environmental adaptability. On the other hand, using the unmanned platform's wheel speed encoders, accelerometers, and other body perception sensors can directly sense the impact of surface attributes, obtaining direct surface feedback as the unmanned platform passes through. This is also insensitive to dynamic environmental changes, lighting, season, and weather, exhibiting strong robustness. However, contact-based perception cannot achieve pre-judgment before the unmanned platform passes through.

[0003] Land attribute classification methods based on vision or LiDAR, land attribute classification based on machine perception, and land attribute classification methods that combine both have all been proposed. However, most of these methods use supervised learning to train static models, which lack the ability to adapt to environmental changes. For example, when faced with environmental changes caused by lighting, or when the same type of land attribute may have different characteristics in complex environments, or when encountering scenes not present in the training set, the classification performance of these methods deteriorates and they lack sufficient robustness. Summary of the Invention

[0004] The purpose of this invention is to provide a training method and system for a visual surface attribute classification prediction model, so as to solve at least one of the technical problems existing in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] On one hand, the present invention provides a method for training a visual surface attribute classification prediction model, comprising:

[0007] Acquire training data; wherein, the training data includes multiple surface images and labels for surface attribute features in the images;

[0008] A visual land surface attribute classification prediction model is trained based on training data. The model includes a visual module, a body perception module, and a classification network. The input of the visual module is a 1-frame image patch, and the output is a land surface attribute label. The input of the body perception module is a 100-millisecond time-series signal from the accelerometer and wheel speed encoder, and the output is a land surface attribute label. The classification network learns the model parameters of the visual module and the body perception module in a fully supervised manner based on manually labeled samples to obtain the land surface attribute classification.

[0009] Optionally, the visual perception-based classification of surface attributes includes: at time t, the unmanned platform uses a forward-facing camera to acquire a surface image patch p of region A in front of it. t Based on the visual features of the surface image patches, the surface attributes of the area A in front are predicted using a visual module.

[0010] Optionally, surface attribute classification based on body perception includes: at time t+1, the unmanned platform passes through region A and collects body perception data of region A through body perception; based on the information perceived by the unmanned platform in contact with the ground, the surface attributes of region A are classified according to the body perception module to obtain classification labels.

[0011] Optionally, if the visual perception result differs from the organism's perception result, and the confidence level of the organism's perception is higher than a threshold, then a sample is constructed. Will monitor loss Backpropagation is performed to optimize the parameters θ of the visual prediction module and improve its environmental adaptability.

[0012] Secondly, the present invention provides a training system for a visual surface attribute classification and prediction model, comprising:

[0013] An acquisition module is used to acquire training data; wherein, the training data includes multiple land surface images and labels of land surface attribute features in the images;

[0014] The training module is used to train the visual land surface attribute classification prediction model based on training data. The visual land surface attribute classification prediction model includes a vision module, a body perception module, and a classification network. The input of the vision module is a 1-frame image patch, and the output is the land surface attribute label. The input of the body perception module is the time-series signal from the accelerometer and wheel speed encoder over 100 milliseconds, and the output is the land surface attribute label. The classification network learns the model parameters of the vision module and the body perception module in a fully supervised manner based on manually labeled samples to obtain the land surface attribute classification.

[0015] Thirdly, the present invention provides a visual surface attribute classification and prediction method, comprising:

[0016] Acquire surface image data;

[0017] The acquired surface image data is processed using a visual surface attribute classification prediction model to obtain surface attribute classification results; wherein the visual surface attribute classification prediction model is trained using the model training method described in the first aspect.

[0018] Fourthly, the present invention provides a visual surface attribute classification and prediction system, comprising:

[0019] The acquisition module is used to acquire surface image data;

[0020] The prediction module is used to process the acquired surface image data using a visual surface attribute classification prediction model to obtain surface attribute classification results; wherein the visual surface attribute classification prediction model is trained using the model training method described in the first aspect.

[0021] Fifthly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the visual surface attribute classification prediction model training method as described above.

[0022] In a sixth aspect, the present invention provides a computer device including a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions executable by the processor, and the processor calling the program instructions to execute the visual surface attribute classification prediction model training method as described above.

[0023] In a seventh aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the visual surface attribute classification prediction model training method as described above.

[0024] Eighthly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the visual surface attribute classification and prediction method as described in the third aspect.

[0025] In a ninth aspect, the present invention provides a computer device including a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions executable by the processor, and the processor calling the program instructions to execute the visual surface attribute classification prediction method described in the third aspect.

[0026] In a tenth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the visual surface attribute classification prediction method as described in the third aspect.

[0027] The beneficial effects of this invention are as follows: Addressing the problems of poor adaptability of visual perception to environmental changes, its susceptibility to influences such as light, seasons, and weather, and its lack of sufficient robustness, this invention utilizes the stability of bodily perception to environmental changes. Supervised samples are constructed using bodily perception results from the same surface, and the visual perception model is optimized online to enable it to adapt to environmental changes and improve its robustness.

[0028] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the online learning method for the visual surface attribute classification and prediction model described in an embodiment of the present invention.

[0031] Figure 2 This is a functional framework diagram of the online learning system for the visual surface attribute classification and prediction model described in an embodiment of the present invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0034] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0035] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0036] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0037] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0038] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0039] Example 1

[0040] In this embodiment 1, a visual land surface attribute classification and prediction system is first provided, including: an acquisition module for acquiring land surface image data; and a prediction module for processing the acquired land surface image data using a visual land surface attribute classification and prediction model to obtain land surface attribute classification results.

[0041] In this embodiment 1, the above system is used to implement a visual land surface attribute classification prediction method, including: acquiring land surface image data; processing the acquired land surface image data using a visual land surface attribute classification prediction model to obtain land surface attribute classification results; wherein, the visual land surface attribute classification prediction model is trained using a visual land surface attribute classification prediction model training method.

[0042] In this embodiment, the training method for the visual surface attribute classification prediction model includes: acquiring training data; wherein the training data includes multiple surface images and labels of surface attribute features in the annotated images; training the visual surface attribute classification prediction model based on the training data, wherein the visual surface attribute classification prediction model includes a visual module, a body perception module, and a classification network; the input of the visual module is a 1-frame image patch, and the output is a surface attribute label; the input of the body perception module is a 100-millisecond time-series signal from the accelerometer and wheel speed encoder, and the output is a surface attribute label; the classification network learns the model parameters of the visual module and the body perception module in a fully supervised manner based on manually labeled samples to obtain the surface attribute classification.

[0043] Visual perception-based surface attribute classification includes: at time t, the unmanned platform uses a forward-facing camera to acquire a surface image patch p of region A in front of it. t Based on the visual features of the surface image patches, the surface attributes of the area A in front are predicted using a visual module.

[0044] The surface attribute classification based on body perception includes: at time t+1, the unmanned platform passes through region A and collects body perception data of region A through body perception; based on the information sensed by the unmanned platform in contact with the ground, the surface attributes of region A are classified according to the body perception module to obtain classification labels.

[0045] When visual perception results differ from bodily perception results, and the confidence level of bodily perception is higher than a threshold, samples are constructed. Will monitor loss Backpropagation is performed to optimize the parameters θ of the visual prediction module and improve its environmental adaptability.

[0046] Example 2

[0047] In this embodiment 2, a training system for a visual land surface attribute classification prediction model is provided, comprising: an acquisition module for acquiring training data; wherein the training data includes multiple land surface images and labels of land surface attribute features in the annotated images; and a training module for training the visual land surface attribute classification prediction model based on the training data, wherein the visual land surface attribute classification prediction model includes a visual module, a body perception module, and a classification network; the input of the visual module is a 1-frame image patch, and the output is a land surface attribute label; the input of the body perception module is a 100-millisecond time-series signal from an accelerometer and a wheel speed encoder, and the output is a land surface attribute label; and the classification network learns the model parameters of the visual module and the body perception module in a fully supervised manner based on manually labeled samples to obtain the land surface attribute classification.

[0048] In this embodiment, to address the problems of poor adaptability of visual perception to environmental changes, its susceptibility to factors such as lighting, seasons, and weather, and its lack of robustness, this embodiment utilizes the stability of bodily perception to environmental changes. Supervised samples are constructed using bodily perception results on the same surface. Based on these samples, the visual perception model is optimized online to enable it to adapt to environmental changes and improve its robustness.

[0049] Online learning methods for visual surface attribute classification and prediction models, such as... Figure 1 As shown, it includes the following steps:

[0050] Offline pre-training: Parameters are learned in a fully supervised manner using manually labeled data to construct visual perception and body perception modules;

[0051] Online processing includes the following three steps.

[0052] Classification of surface attributes based on visual perception;

[0053] Classification of surface attributes based on organismal perception;

[0054] When visual perception and bodily perception results differ, and bodily perception has high confidence, online visual perception model optimization is performed.

[0055] like Figure 2 As shown, the specific steps are as follows.

[0056] Model selection and offline pre-training;

[0057] A CNN architecture is selected to establish the vision module f and the body perception module g. The deep network architecture selected in this invention includes, but is not limited to, the CNN architecture. The input of module f is a 1-frame image patch, and the output is the surface attribute label. The input of module g is the 100-millisecond temporal signal from the accelerometer and wheel speed encoder, and the output is the surface attribute label. Manually labeled samples are used to learn the model parameters of f and g in a fully supervised manner to construct a surface attribute classifier.

[0058] The following are the online processing steps.

[0059] Surface attribute classification based on forward-looking and visual perception: At time t, the unmanned platform uses its forward-facing camera to acquire surface image patch p of region A in front of it. t Based on p t Based on the visual features, and using visual module f, the surface attributes of the area A in front are predicted.

[0060] Based on surface attribute classification using machine perception: At time t+1, the unmanned platform passes through region A and collects machine perception data v of region A using machine perception. t+1 Based on vt+1 The information sensed by the unmanned platform upon contact with the ground, based on the body perception module g, is used to classify the surface attributes of region A and obtain classification labels. The softmax confidence score output by the classification network is used to obtain... The confidence evaluation index selected in this embodiment includes, but is not limited to, the Softmax confidence level.

[0061] Online optimization of visual perception models.

[0062] Visual perception results differ from bodily perception results, that is... And the body's perception confidence level is higher than the threshold, that is At that time, construct samples Will monitor loss By performing backpropagation, the parameters θ of the visual prediction module f can be optimized online, improving its environmental adaptability.

[0063] Example 3

[0064] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they implement the visual surface attribute classification and prediction method described above.

[0065] Example 4

[0066] This embodiment 4 provides a computer device, including a memory and a processor. The processor and the memory communicate with each other. The memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute a visual surface attribute classification prediction method.

[0067] Example 5

[0068] This embodiment 4 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions to implement the visual surface attribute classification prediction method as described above.

[0069] In summary, the visual surface attribute classification prediction model training method and system described in this embodiment of the invention utilizes the stability of organism perception in response to environmental changes, constructs supervised samples using the organism perception results of the same surface, and optimizes the visual perception model online on this basis, enabling it to adapt to environmental changes and improving its robustness.

[0070] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A method for training a visual surface attribute classification and prediction model, characterized in that, include: Acquire training data; wherein, the training data includes multiple surface images and labels for surface attribute features in the images; A visual land surface attribute classification prediction model is trained based on training data. This model includes a visual module, a machine perception module, and a classification network. The visual module takes a single frame of image data as input and outputs land surface attribute labels. The machine perception module takes 100 milliseconds of accelerometer and wheel speed encoder time-series signals as input and outputs land surface attribute labels. The classification network learns the model parameters of the visual and machine perception modules in a fully supervised manner based on manually labeled samples to obtain land surface attribute classifications. Specifically, the land surface attribute classification based on visual perception includes: at time t, the unmanned platform uses its forward-facing camera to capture a land surface image patch of region A in front of it. Based on the visual features of surface image patches, and using a visual module, the surface attributes of region A ahead are predicted. The surface attribute classification based on the unmanned platform's perception includes: at time t+1, the unmanned platform passes through region A and collects perception data of region A through its perception capabilities; based on the information perceived by the unmanned platform upon contact with the ground, and using the perception module, the surface attributes of region A are classified to obtain classification labels. When visual perception results differ from bodily perception results, and the confidence level of bodily perception is higher than a threshold, a sample is constructed. , >, monitor losses Perform backpropagation to optimize the parameters of the visual prediction module. This improves its environmental adaptability.

2. A training system for a visual surface attribute classification and prediction model based on the method described in claim 1, characterized in that, include: An acquisition module is used to acquire training data; wherein, the training data includes multiple land surface images and labels of land surface attribute features in the images; The training module is used to train the visual land surface attribute classification prediction model based on training data. The visual land surface attribute classification prediction model includes a vision module, a body perception module, and a classification network. The input of the vision module is a 1-frame image patch, and the output is the land surface attribute label. The input of the body perception module is the time-series signal from the accelerometer and wheel speed encoder over 100 milliseconds, and the output is the land surface attribute label. The classification network learns the model parameters of the vision module and the body perception module in a fully supervised manner based on manually labeled samples to obtain the land surface attribute classification.

3. A method for classifying and predicting visual surface attributes, characterized in that, include: Acquire surface image data; The acquired surface image data is processed using a visual surface attribute classification prediction model to obtain surface attribute classification results; wherein the visual surface attribute classification prediction model is trained using the model training method described in claim 1.

4. A visual surface attribute classification and prediction system, characterized in that, include: The acquisition module is used to acquire surface image data; The prediction module is used to process the acquired surface image data using a visual surface attribute classification prediction model to obtain surface attribute classification results; wherein the visual surface attribute classification prediction model is trained using the model training method described in claim 1.

5. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the visual surface attribute classification and prediction method as described in claim 3.

6. A computer device, characterized in that, It includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions that can be executed by the processor, and the processor calling the program instructions to execute the visual surface attribute classification prediction method as described in claim 3.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the visual surface attribute classification prediction method as described in claim 3.

Images