Image segmentation method, system and device and storage medium

Abstract

The embodiment of the invention provides an image segmentation method, system and device and a storage medium, and the method comprises the steps: obtaining a to-be-segmented image which comprises non-ground semantic elements; determining an interested object based on the to-be-segmented image and a trained semantic segmentation model, wherein the semantic segmentation model is a machine learning model; determining point cloud depth information corresponding to the to-be-segmented image based on the to-be-segmented image and the point cloud data; and determining a target mask of the to-be-segmented image based on the object of interest and the point cloud depth information.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to an image segmentation method, system, device and storage medium. Background Art

[0002] Autonomous driving is a way of driving that is achieved through a processor-based automated system without human real-time operation. Autonomous vehicles rely on the collaboration of artificial intelligence, visual computing, detection radar, monitoring devices and positioning systems to enable the system to automatically and safely control vehicles without human real-time operation.

[0003] For autonomous vehicles, it is very important to be able to accurately perceive the surrounding traffic environment. With the rapid development of deep learning technology, semantic segmentation has become an indispensable part in the field of autonomous driving environment perception. Semantic segmentation requires the classification of objects pixel by pixel, dividing a given image into several visually meaningful or interesting areas to facilitate subsequent image analysis and visual understanding. However, due to the richness and diversity of real-world scenes, semantic segmentation has become a basic and challenging task. For example, objects such as traffic signs and poles are affected by size and the distance of the scene, and their recognition is low, and the problem of false detection of target edges is relatively common. Furthermore, if the semantic segmentation results are directly used as downstream input, there will be a lot of noise points, which makes it difficult to meet its further vectorization needs.

[0004] Therefore, it is hoped to provide an image segmentation method, system, device and storage medium, which can obtain segmentation results more accurately and improve the recognition of semantic segmentation. Summary of the invention

[0005] In order to improve the recognition of semantic segmentation and accurately filter out the edge noise of semantic elements, so as to accurately obtain the required semantic elements and realize auxiliary positioning, the present invention provides an image segmentation method, system, device and storage medium, which can efficiently and accurately perform operations such as perception, positioning, prediction, decision-making and planning of autonomous driving.

[0006] One or more embodiments of the present specification provide an image segmentation method, the method comprising: acquiring an image to be segmented, the image to be segmented comprising non-ground semantic elements; determining an object of interest based on the image to be segmented and a trained semantic segmentation model, the semantic segmentation model being a machine learning model; determining point cloud depth information corresponding to the image to be segmented based on the image to be segmented and point cloud data; and determining a target mask of the image to be segmented based on the object of interest and the point cloud depth information.

[0007] In some embodiments, based on the object of interest and the point cloud depth information, determining the target mask of the image to be segmented includes: performing an AND operation based on a semantic mask corresponding to the object of interest and a point cloud mask corresponding to the point cloud depth information to obtain a first operation mask; dilating the first operation mask to obtain a second operation mask; and performing a histogram analysis and a voting operation on the second operation mask to determine the target mask.

[0008] In some embodiments, the non-ground semantic elements include at least one or more of traffic signs, traffic lights, and fences.

[0009] In some embodiments, the method further includes: correcting the position of the autonomous driving vehicle based on the target mask; and performing autonomous driving control based on the corrected position.

[0010] In some embodiments, determining the point cloud depth information of the image to be segmented includes: in response to a ratio of a first amount of point cloud depth information to a second amount of image pixel point information satisfying a preset condition, fusing multiple frames of the point cloud data to obtain a third amount of point cloud depth information, wherein the third amount is determined based on the number of objects of interest.

[0011] In some embodiments, the semantic segmentation model is obtained by: acquiring a sample image; annotating the sample image based on the mask category of the non-ground semantic element corresponding to each pixel in the sample image; inputting the annotated sample image into an initial semantic segmentation model for training to determine the semantic segmentation model.

[0012] In some embodiments, determining the point cloud depth information of the image to be segmented based on the image to be segmented and the point cloud data includes: mapping the point cloud data to the image to be segmented based on a coordinate conversion relationship between an image coordinate system and a lidar coordinate system, and determining the point cloud depth information.

[0013] One or more embodiments of the present specification provide an image segmentation system, the system comprising: an acquisition module, used to acquire an image to be segmented, the image to be segmented comprising non-ground semantic elements; a segmentation module, used to determine an object of interest based on the image to be segmented and a trained semantic segmentation model, the semantic segmentation model being a machine learning model; an analysis module, used to determine point cloud depth information of the image to be segmented based on the image to be segmented and point cloud data; and a determination module, used to determine a target mask of the image to be segmented based on the object of interest and the point cloud depth information.

[0014] One or more embodiments of the present specification provide an image segmentation device, which includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least part of the computer instructions to implement the image segmentation method as described above.

[0015] One or more embodiments of the present specification provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the image segmentation method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0017] Figure 1 is an exemplary schematic diagram of an application scenario of an image segmentation system according to some embodiments of this specification;

[0018] Figure 2 is an exemplary module diagram of an image segmentation system according to some embodiments of this specification;

[0019] Figure 3 is an exemplary flow chart of an image segmentation method according to some embodiments of this specification;

[0020] Figure 4 is an exemplary schematic diagram of a target mask determination method according to some embodiments of this specification;

[0021] Figure 5 It is an exemplary schematic diagram of the training and application of the semantic segmentation model shown in some embodiments of this specification. DETAILED DESCRIPTION

[0022] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0023] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0024] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0025] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0026] Figure 1 It is an exemplary schematic diagram of the application scenario of the image segmentation system shown in some embodiments of this specification. In some embodiments, the system and method in this application can be applied to scenarios of terminal mobile control, such as automatic driving scenarios, such as pure unmanned driving scenarios without drivers, assisted driving modules in vehicles, and other scenarios. The assisted driving module in the vehicle can autonomously control the vehicle's driving, and can also output driving prompts to the driver. The movable terminal can be applied to land, sea, aerospace, etc., or any combination thereof. The movable terminal may include taxis, private cars, ride-sharing, shared vehicles, buses, trains, motor vehicles, high-speed railways, subways, ships, airplanes, spacecraft, hot air balloons, bicycles, tricycles, motorcycles, robots, etc., or any combination thereof. In some embodiments, the image segmentation system can realize mobile control of the terminal by implementing the methods and / or processes disclosed in this specification, for example, performing operations such as perception, positioning, prediction, decision-making and planning related to the movement of the terminal. In some embodiments, the image segmentation system can accurately obtain non-ground semantic elements by implementing the methods and / or processes disclosed in this specification, thereby assisting in positioning.

[0027] In some embodiments, Figure 1 As shown, the application scenario 100 of the image segmentation system may include a processing device 110 , a terminal 120 , a user terminal 130 , a storage device 140 and a network 150 .

[0028] The processing device 110 can obtain and / or process various information and data. In some embodiments, the processing device 110 can perform the image segmentation method as shown in some embodiments of this specification. For example, the processing device 110 can obtain an image to be segmented, and process the image to be segmented based on a trained semantic segmentation model to determine an object of interest. For another example, the processing device 110 can determine the point cloud depth information corresponding to the image to be segmented based on the image to be segmented and the point cloud data, and determine the target mask of the image to be segmented based on the object of interest and the point cloud depth information.

[0029] In some embodiments, the processing device 110 may include a central processing unit (CPU), a digital signal processor (DSP), a system on a chip (SoC), a microcontroller unit (MCU), etc. and / or any combination thereof. In some embodiments, the processing device 110 may include a computer, a user console, a single server or a server group, etc. The server group may be centralized or distributed. In some embodiments, the processing device 110 may be local or remote. For example, the processing device 110 may access information and / or data stored in the storage device 140 via the network 150. For another example, the processing device 110 may directly connect the storage device 140 to access the stored information and / or data. In some embodiments, the processing device 110 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc. or any combination thereof. In some embodiments, the processing device 110 or a portion of the processing device 110 may be integrated into the terminal 120.

[0030] The terminal 120 may be an object controlled by the image segmentation system. The terminal 120 may be applied to land, sea, aerospace, etc., or any combination thereof. The terminal 120 may include taxis, private cars, rideshares, shared vehicles, buses, trains, motor vehicles, high-speed railways, subways, ships, airplanes, spacecrafts, hot air balloons, bicycles, tricycles, motorcycles, robots, etc., or any combination thereof. In some embodiments, the terminal 120 may receive instructions issued by the processing device 110 and complete corresponding tasks in accordance with the instructions. For example, the terminal 120 may receive instructions for executing positioning corrections, driving operations, etc., and automatically control the terminal 120 to complete the corresponding operations.

[0031] In some embodiments, the terminal 120 may include a control component, a positioning component (e.g., a GPS positioning module), a timing component (e.g., a timer), a sensor component (e.g., a motion sensor, a pressure sensor, a radar sensor, an infrared sensor, and an image acquisition sensor), a communication component (e.g., a GPS communication module, a short-range wireless communication module, etc.), etc. The positioning component is mainly used to locate the terminal 120. The sensor component is mainly used to collect sensor data (e.g., image data, laser point cloud data, speed, temperature, etc.) during the movement of the terminal 120. In some embodiments, the terminal 120 can be autonomously controlled under the coordinated action of the control component, the positioning component, the timing component, the sensor component, and the communication component. For example, the terminal 120 can generate instructions such as automatic driving operations and perform autonomous control under the coordinated action of the control component, the positioning component, the timing component, the sensor component, and the communication component.

[0032] In some embodiments, the terminal 120 may communicate with at least one component of the image segmentation system through the network 150. For example, the terminal 120 may send the current location information of the terminal 120 to the processing device 110 and / or the user terminal 130 through the network 150. In some embodiments, the terminal 120 may communicate with the server 110 through the network 150. For example, the terminal 120 may send the current location information of the vehicle to the processing device 110 and / or the user terminal 130 through the network 150.

[0033] The user terminal 130 can realize the interaction between the user and the image segmentation system. In some embodiments, the user terminal 130 can be a terminal device used by the user when using the vehicle service. In some embodiments, the user terminal 130 can display the current path planning of the terminal 120 to the user. In some embodiments, the user terminal 130 can include a mobile device, a tablet computer, a laptop computer, other devices with input and / or output functions, etc. or any combination thereof.

[0034] The storage device 140 may be used to store data, instructions and / or any other information. In some embodiments, the storage device 140 may store data and / or information processed by the processing device 110, such as point cloud data, target mask, etc. The storage device 140 may include one or more storage components, each of which may be an independent device or part of other components in the application scenario 100 (e.g., the processing device 110, the terminal 120). In some embodiments, the storage device 140 may include a random access memory (RAM), a read-only memory (ROM), a mass storage device, a removable memory, a volatile read-write memory, etc., or any combination thereof. In some embodiments, the storage device 140 may be implemented on a cloud platform. In some embodiments, the storage device 140 may be integrated or included in one or more other components of the application scenario 100 (e.g., the processing device 110, the terminal 120, or other possible components).

[0035] The network 150 can connect the components of the system and / or connect the system with external resources. The network 150 enables communication between the components and with other parts outside the system to facilitate the exchange of data and / or information. In some embodiments, one or more components of the application scenario 100 (e.g., the processing device 110, the terminal 120, the storage device 140) can send information and / or data to other components through the network 150. In some embodiments, the network 150 may include any one or more of a wired network or a wireless network. In some embodiments, the network 150 may be a point-to-point, shared, centralized, or other topological structure or a combination of multiple topological structures. In some embodiments, the network 150 may include one or more network access points. For example, the network 150 may include a wired or wireless network access point, through which one or more components of the application scenario 100 may be connected to the network 150 to exchange data and / or information.

[0036] It should be noted that application scenario 100 is provided only for the purpose of illustration and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various modifications or changes can be made according to the description of this specification. For example, application scenario 100 can also include a database. For another example, application scenario 100 can be implemented on other devices to achieve similar or different functions. However, changes and modifications will not deviate from the scope of this specification.

[0037] Figure 2 It is an exemplary module diagram of an image segmentation system according to some embodiments of the present specification.

[0038] like Figure 2As shown, in some embodiments, the image segmentation system 200 may include an acquisition module 210, a segmentation module 220, an analysis module 230, and a determination module 240. In some embodiments, each module in the image segmentation system 200 may be implemented by the processing device 110.

[0039] In some embodiments, the acquisition module 210 may be used to acquire the image to be segmented. For more information about the image to be segmented, see Figure 3 and related instructions.

[0040] In some embodiments, the segmentation module 220 can be used to determine the object of interest based on the image to be segmented and the trained semantic segmentation model. For more information about the semantic segmentation model and determining the object of interest, please refer to Figure 3 , Figure 5 and related instructions.

[0041] In some embodiments, the analysis module 230 may be configured to determine point cloud depth information of the image to be segmented based on the image to be segmented and the point cloud data.

[0042] In some embodiments, in response to a ratio of a first amount of point cloud depth information to a second amount of image pixel point information satisfying a preset condition, the analysis module 230 may also be used to fuse multiple frames of point cloud data to obtain a third amount of point cloud depth information.

[0043] In some embodiments, the analysis module 230 may also map the point cloud data to the image to be segmented based on the coordinate conversion relationship between the image coordinate system and the lidar coordinate system to determine the point cloud depth information.

[0044] For more information about point cloud data, point cloud depth information, and how to obtain point cloud depth information, see Figure 3 and related instructions.

[0045] In some embodiments, the determination module 240 may be configured to determine a target mask of the image to be segmented based on the object of interest and the point cloud depth information.

[0046] In some embodiments, the determination module 240 may also be used to perform an AND operation based on the semantic mask corresponding to the object of interest and the point cloud mask corresponding to the point cloud depth information to obtain a first operation mask; dilate the first operation mask to obtain a second operation mask; and perform histogram analysis and voting operations on the second operation mask to determine a target mask. For more information about the target mask and how to determine the target mask, please refer to Figure 3 , Figure 4 and related instructions.

[0047] In some embodiments, the image segmentation system 200 may further include an autonomous driving module ( Figure 2not shown).

[0048] In some embodiments, the autonomous driving module can correct the position of the autonomous driving vehicle based on the target mask and perform autonomous driving control based on the corrected position. For more information about position correction and autonomous driving control, see Figure 3 and its related description.

[0049] It should be noted that the above description of the image segmentation system 200 and its modules is only for convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that, after understanding the principle of the device, those skilled in the art may arbitrarily combine the modules or form sub-devices to connect with other modules without deviating from the principle. In some embodiments, Figure 2 The modules disclosed in the specification may be different modules in a system, or one module may realize the functions of two or more modules. For example, each module may share a storage module, or each module may have its own storage module. Such variations are within the scope of protection of this specification.

[0050] Figure 3 is an exemplary flow chart of an image segmentation method according to some embodiments of the present specification. In some embodiments, process 300 can be executed by processing device 110 or image segmentation system 200. For example, process 300 can be stored in the storage device 140 in the form of a program or instruction, and when processing device 110 or image segmentation system 200 executes the instruction, process 300 can be implemented. The operational diagram of process 300 presented below is illustrative. In some embodiments, the process can be completed using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 3 The order in which the operations of process 300 are shown and described below is not limiting. Figure 3 As shown, process 300 includes the following steps.

[0051] Step 310 , obtaining the image to be segmented. In some embodiments, step 310 may be performed by the obtaining module 210 .

[0052] The image to be segmented refers to an image that needs to be semantically segmented. For example, the image to be segmented may be image data related to the environment surrounding the terminal. In some embodiments, the image to be segmented includes at least non-ground semantic elements.

[0053] Non-ground semantic elements refer to content that is not related to the road surface. Non-ground semantic elements can be static. For example, objects fixed on the road and its surroundings, including poles (e.g., street light poles, traffic light poles, etc.), walls (e.g., buildings on both sides of the road, etc.), green vegetation, etc.

[0054] In some embodiments, the non-ground semantic elements include at least one or more of traffic signs, traffic lights, and fences. Traffic signs include but are not limited to post signs, cantilever signs, temporary signs, etc. Traffic lights include but are not limited to post traffic lights and temporary traffic lights. Fences include but are not limited to railings, stone piers, and isolation fences located on both sides of the road.

[0055] The above descriptions of non-ground semantic elements are for illustrative purposes only and are not intended to limit the scope of this specification.

[0056] In some embodiments, the acquisition module 210 may acquire images to be segmented by an image acquisition component, wherein the image acquisition component includes but is not limited to a front view camera, a surround view camera, a side view camera, a rear view camera, etc. installed on a terminal (e.g., a vehicle).

[0057] In some embodiments, the acquisition module 210 can acquire images within a preset range through an image acquisition component to acquire the image to be segmented. The preset range can be in various forms. In some embodiments, the preset range can be at least one of a range of a certain distance in front of the terminal, a range of a certain distance to the side of the terminal, a range of a certain distance behind the terminal, and a range of a certain distance above the terminal. The preset range can be set based on actual needs and is not limited here.

[0058] In some embodiments, the acquisition module 210 can control the image acquisition component to perform multiple image acquisitions during the terminal's driving process to acquire the image to be segmented. For example, the acquisition module 210 can control the terminal to pass through the same road section multiple times, and perform image acquisition each time the terminal passes through the road section to acquire the image to be segmented.

[0059] In some embodiments, the acquisition module 210 can acquire images through an image acquisition component to acquire raw image data, and preprocess the raw image data to obtain the image to be segmented. The preprocessing of the raw image data includes but is not limited to one or more of image filtering, image denoising, image equalization, image enhancement, etc. Through preprocessing, the light of the image can be improved, the influence of strong light glare can be removed, the light of the image can be balanced, and the key information can be highlighted.

[0060] Step 320 , determining the object of interest based on the image to be segmented and the trained semantic segmentation model. In some embodiments, step 320 may be performed by the segmentation module 220 .

[0061] The object of interest refers to an object with a semantic label obtained after semantic segmentation. For example, the object of interest can be a non-ground semantic element such as a traffic light or a traffic sign pole that is annotated with a semantic label.

[0062] like Figure 5 As shown, in some embodiments, the segmentation module 220 may input the image to be segmented 510 into a trained semantic segmentation model 520, and output an object of interest 530. In some embodiments, the semantic segmentation model 520 is a machine learning model.

[0063] In some embodiments, the segmentation module 220 may utilize an existing semantic segmentation model 520 to process the image to be segmented 510 and determine an object of interest 530 .

[0064] In some embodiments, the semantic segmentation model 520 includes at least 800 or more multiplication operations in one execution; in some embodiments, the semantic segmentation model 520 is at least partially executed by a GPU.

[0065] In some embodiments, the existing semantic segmentation model 520 includes but is not limited to a combination of one or more of a Fully Convolutional Networks (FCN) model, a Pyramid Scene Parseing Network (PSP Net) model, an Efficient Neural Network (ENet) model and / or a Mask Region-based Convolutional Neural Network (Mask R-CNN) model.

[0066] In some embodiments, the semantic segmentation model 520 may be trained in advance by the processing device 110 and stored in the storage device 140 .

[0067] See also Figure 5 In some embodiments, the segmentation module 220 can obtain a sample image 540, and annotate the sample image 540 based on the mask category 550 of the non-ground semantic element corresponding to each pixel in the sample image 540; input the annotated sample image 560 into the initial semantic segmentation model 570 for training, and determine the trained semantic segmentation model 520.

[0068] In some embodiments, the initial semantic segmentation model 570 may be an existing semantic segmentation model or an untrained machine learning model.

[0069] In some embodiments, the segmentation module 220 can obtain sample images based on image data historically collected by the image acquisition component. For example, the segmentation module 220 can use the image data collected historically as sample images. For another example, the segmentation module 220 can preprocess the image data collected historically and use the preprocessed image data as sample data. In some embodiments, the segmentation module 220 can use existing datasets as sample images. Exemplary existing datasets include one or more combinations of PascalVOC 2012, Cityscapes, Pascal Context, and Stanford Background Dataset.

[0070] In some embodiments, the segmentation module 220 can determine various objects contained in the sample image 540 based on object recognition technology; filter out objects that do not belong to non-ground semantic elements, and annotate the pixels corresponding to the remaining objects with mask categories 550 according to the recognition results to obtain annotated sample images 560. Exemplary object recognition technologies include, but are not limited to, one or more combinations of histogram of gradients (HOG features), regional convolutional neural networks (R-CNN), spatial pyramid pooling networks (SPP-net), regression-based object detectors, and the like.

[0071] In some embodiments, the mask category of the non-ground semantic element corresponding to each pixel in the sample image 540 may also be manually determined based on prior knowledge annotation.

[0072] An exemplary training process: input the sample image 540 into the initial semantic segmentation model 570 to obtain the segmentation data output by the initial semantic segmentation model 570, i.e., the mask category of the non-ground semantic element corresponding to each pixel in the sample image 540 output by the initial semantic segmentation model 570; construct a loss function based on the segmentation data and the labeled sample image 560, update the model parameters of the initial semantic segmentation model 570, and continue training until the preset conditions are met to obtain the desired model, i.e., the trained semantic segmentation model 520. The preset conditions may include one or more of the loss function being less than a threshold, convergence, or the training cycle reaching a threshold.

[0073] In some embodiments of the present specification, a trained semantic segmentation model is used to process images to be segmented. The self-learning ability of the machine learning model can be used to find patterns from a large amount of image data, obtain the correlation between non-ground semantic elements and image data, improve the accuracy and efficiency of determining non-ground semantic elements, and segment objects of interest more quickly and accurately.

[0074] Step 330 , based on the image to be segmented and the point cloud data, determine the point cloud depth information corresponding to the image to be segmented. In some embodiments, step 330 may be performed by the analysis module 230 .

[0075] As used herein, point cloud data may include a set of data points associated with one or more objects in the surrounding environment of an object (e.g., a terminal). The surrounding environment of an object may refer to the environment around the object and one or more objects (including biological and non-biological objects). The surrounding environment may cover an area of ​​any size and shape. In some embodiments, the area covered by the surrounding environment may be related to the performance of a sensor component (e.g., a lidar) mounted on the object. Taking an autonomous driving vehicle traveling on a lane as an example, the surrounding environment of the autonomous driving vehicle may include one or more objects around the autonomous driving vehicle, such as the ground, lane markings, buildings, pedestrians, animals, plants, one or more other vehicles, etc. The size of the area covered by the surrounding environment of the autonomous driving vehicle may depend on (or partially depend on) the scanning range of the lidar installed on the autonomous driving vehicle.

[0076] The data points of the point cloud data may correspond to physical points or areas of an object in the space around the estimated position of the object (e.g., a terminal). One or more objects around the object may include lane markings, buildings, pedestrians, animals, plants, vehicles, etc. In some embodiments, the point cloud data may be acquired by a sensor assembly mounted on the object. For example, a sensor may emit a laser pulse to scan the surroundings of the object. The laser pulse may be reflected by a physical point in the surroundings and returned to the sensor. The sensor may generate point cloud data representing the surroundings based on one or more features of the returned laser pulse. In some embodiments, point cloud data may be collected within a time period (e.g., 1 second, 2 seconds) during which an object (e.g., vehicle 110) is parked on a road or traveling along a road. During the collection of point cloud data, the sensor may rotate within a scanning angle range (e.g., 360 degrees, 180 degrees, 120 degrees) and scan the surroundings at a specific scanning frequency (e.g., 10Hz, 15Hz, 20Hz).

[0077] In some embodiments, the point cloud data may include at least one feature value of at least one feature of the physical point. Exemplary features of the physical point may include the location of the physical point, the intensity of the physical point, the classification of the physical point, the scanning direction associated with the physical point, etc., or any combination thereof. In some embodiments, the location of the physical point may include the geographical location of the physical point. For example, the location of the physical point may be expressed as latitude, longitude and / or altitude in a geographic coordinate system. In some embodiments, the location of the physical point may be a relative position relative to the sensor assembly (or object). In some embodiments, the relative position of the physical point relative to the sensor assembly (or object) may be expressed as the coordinates of the physical point in a reference coordinate system associated with the sensor assembly (or object). The reference coordinate system associated with the sensor assembly (or object) may be any suitable coordinate system, the origin of which is located at the sensor assembly (or object). By way of example only, the reference coordinate system may be a three-dimensional coordinate system having an X-axis, a Y-axis and a Z-axis. The Y-axis may be parallel to the direction of the object. The X-axis may be perpendicular to the Y-axis and form an XY plane that is parallel or substantially parallel to the ground surface or bottom surface of the object. The Z-axis may be perpendicular to the XY plane. The coordinates of the physical point in the reference coordinate system may include one or more of an X coordinate on the X axis, a Y coordinate on the Y axis, and a Z coordinate on the X axis, wherein the Z coordinate may also be referred to as a relative height of the physical point relative to the sensor.

[0078] The intensity information of a physical point may refer to the return intensity of a laser pulse emitted from a sensor component (e.g., LiDAR) and reflected by the physical point, used to generate a data point corresponding to the physical point. The classification of the physical point may refer to the type of physical point (or object) that reflected the laser pulse. For example, the physical point may be classified into many categories, including bare ground or ground, buildings, people, water, etc. The scanning direction associated with the physical point may refer to the direction in which the scanning mirror of the sensor is pointing when the sensor detects the corresponding data point.

[0079] In some embodiments, the point cloud data may be acquired through a sensor assembly. In some embodiments, the point cloud data may be acquired through one or more of a laser radar, a depth camera, a binocular camera, etc. For example, a laser radar may be used to scan the space where the terminal is located to obtain point cloud data.

[0080] In some embodiments, the analysis module 230 can collect point cloud data within a preset range through the sensor component. For more information about the preset range, please refer to the above related description.

[0081] In some embodiments, the analysis module 230 may control the sensor assembly to perform multiple point cloud acquisitions during the terminal's driving process to obtain point cloud data. For example, the analysis module 230 may control the terminal to pass through the same road section multiple times, and perform point cloud acquisition each time the terminal passes through the road section to obtain point cloud data.

[0082] In some embodiments, the analysis module 230 can collect point cloud data through the sensor component, obtain raw point cloud data, and pre-process the raw point cloud data to obtain point cloud data. The pre-processing of the raw point cloud data includes but is not limited to one or more of point cloud denoising, point cloud filtering, point cloud stitching, point cloud registration, point cloud segmentation, etc.

[0083] In some embodiments, the analysis module 230 can acquire point cloud data through different sensor components, obtain multi-modal raw point cloud data, and perform data fusion on the raw point cloud data to obtain point cloud data. For example, image data acquired by the image acquisition component, laser radar data acquired by the laser radar, millimeter wave radar data acquired by the millimeter wave radar, binocular depth data acquired by the binocular camera, etc. can be fused.

[0084] Through point cloud data preprocessing, noise and outliers in the point cloud can be effectively removed, and point cloud data simplification can be achieved while maintaining geometric features, providing a robust data foundation for subsequent depth information alignment.

[0085] Point cloud depth information refers to the depth value information included in each pixel in the image to be segmented. As used herein, the depth value is the distance information of the physical point (or object) corresponding to the pixel relative to the sensor component (or object).

[0086] In some embodiments, the analysis module 230 may use the position information of the physical point in the point cloud data (eg, the relative position of the physical point to the sensor component or the object) as the point cloud depth information.

[0087] In some embodiments, in response to a ratio of a first amount of point cloud depth information to a second amount of image pixel point information satisfying a preset condition, the analysis module 230 may fuse multiple frames of point cloud data to obtain a third amount of point cloud depth information.

[0088] The first quantity refers to the number of point clouds containing depth information in the point cloud data initially collected by the sensor assembly. In some embodiments, the initially collected point cloud data refers to the point cloud data acquired when the vehicle passes a certain road section for the first time. The first quantity of point cloud depth information can be determined by statistics. The description of the initially collected point cloud data is for illustrative purposes only and is not intended to limit the scope of this specification.

[0089] The second number refers to the number of pixels in the image to be segmented. The second number of image pixel information can be determined based on the resolution of the image acquisition component.

[0090] It should be noted that due to insufficient sampling rate, fast LiDAR scanning speed, long distance between physical points or physical points being blocked, the initially collected point cloud data may be highly sparse, that is, some voxel spaces may not have corresponding point clouds. When the number of point clouds in the initially collected point cloud data is insufficient, point cloud completion can be performed by fusing multiple frames of point cloud data.

[0091] The preset condition refers to the condition for determining whether it is necessary to fuse multiple frames of point cloud data. In some embodiments, the preset condition may be that the ratio of the first number to the second number is less than a preset ratio threshold. The preset ratio threshold may be a system default value, an empirical value, a manually preset value, or any combination thereof, and may be set according to actual needs, and this specification does not limit this.

[0092] In some embodiments, in response to the ratio of the first amount of point cloud depth information to the second amount of image pixel point information being less than a preset ratio threshold, the analysis module 230 may fuse multiple frames of point cloud data to obtain a third amount of point cloud depth information. The multiple frames of point cloud data may include point cloud data acquired by multiple point cloud acquisitions during the terminal's driving process.

[0093] The third number refers to the number of point clouds containing depth information in the point cloud data obtained after fusing multiple frames of point cloud data. The third number is greater than the first number.

[0094] In some embodiments, the third number can be determined based on the number of objects of interest. For example, the more objects of interest, the larger the third number. In some embodiments, the third number can be determined based on the number of pixels of the object of interest. For example, the number of pixels of the object of interest (or the total number of pixels of multiple objects of interest) can be set to the third number. For another example, the product of the number of pixels of the object of interest (or the total number of pixels of multiple objects of interest) and a preset coefficient (for example, any value between 0-1) can be set to the third number. The preset coefficient can be determined based on prior knowledge or historical data.

[0095] In some embodiments, the analysis module 230 may fuse multiple frames of point cloud data with the same modality. For example, multiple frames of laser radar data collected by a laser radar may be fused. In some embodiments, the analysis module 230 may fuse multiple frames of point cloud data with different modalities. For example, image data collected by an image acquisition component, multiple frames of laser radar data collected by a laser radar, multiple frames of millimeter wave radar data collected by a millimeter wave radar, multiple frames of binocular depth data collected by a binocular camera, etc. may be fused.

[0096] In some embodiments, the analysis module 230 may fuse multiple frames of point cloud data through a point cloud fusion algorithm. Exemplary point cloud fusion algorithms include, but are not limited to, one or more combinations of feature registration-based, neighbor search-based, ICP (Iterative Closest Point)-based, and superimposition-based fusion algorithms.

[0097] In some embodiments of the present specification, in response to a ratio of a first amount of point cloud depth information to a second amount of image pixel point information satisfying a preset condition, it can be determined that the point cloud depth information is highly sparse, and by fusing multiple frames of point cloud data, richer and denser point cloud depth information can be intuitively and effectively acquired, thereby providing more comprehensive information, which helps to quickly and accurately perform image segmentation.

[0098] In some embodiments, the analysis module 230 may map the point cloud data to the image to be segmented based on the coordinate conversion relationship between the image coordinate system and the lidar coordinate system, and determine the point cloud depth information.

[0099] The image coordinate system refers to a coordinate system established based on the position of the image acquisition component. For example, the image coordinate system can be a three-dimensional coordinate system with an X-axis, a Y-axis, and a Z-axis whose origin is located at the image acquisition component (e.g., a camera).

[0100] The laser radar coordinate system refers to a coordinate system established based on the position of the laser radar. For example, the laser radar coordinate system may be a three-dimensional coordinate system with an X-axis, a Y-axis, and a Z-axis, the origin of which is located at the laser radar.

[0101] Coordinate transformation refers to the transformation relationship between different coordinate expressions of the same physical point in different coordinate systems.

[0102] In some embodiments, the coordinate transformation relationship may be predetermined.

[0103] In some embodiments, the analysis module 230 can determine the coordinate axis directions and units of the image coordinate system and the laser radar coordinate system, and determine the coordinate transformation relationship according to the coordinate transformation formula. The above description of determining the coordinate transformation relationship is only for illustrative purposes and is not intended to limit the scope of this specification.

[0104] In some embodiments, the analysis module 230 can project the point cloud data to the corresponding coordinates of each pixel point of the image to be segmented in the image coordinate system based on the coordinate transformation relationship, and map the depth information contained in the point cloud data to the corresponding coordinates of each pixel point to obtain the point cloud depth information.

[0105] In some embodiments of the present specification, based on the coordinate conversion relationship between the image coordinate system and the laser radar coordinate system, the point cloud data is mapped to the image to be segmented, which helps to quickly and accurately determine the point cloud depth information corresponding to each pixel point of the image to be segmented.

[0106] Step 340 , based on the object of interest and the point cloud depth information, determine a target mask of the image to be segmented. In some embodiments, step 340 may be performed by the determination module 240 .

[0107] The target mask refers to the mask corresponding to the object of interest in the image to be segmented. Different objects of interest may correspond to different target masks. The mask may indicate a partial image region with a semantic label obtained by semantic segmentation.

[0108] In some embodiments, the determination module 240 may determine the target mask of the image to be segmented based on the point cloud depth information and the image region to which the object of interest belongs. In some embodiments, the determination module 240 may determine the depth information distribution of the image region to which the object of interest belongs based on the point cloud depth information; and determine the target mask of the image to be segmented according to the depth information distribution and the preset distribution condition. When the depth information distribution meets the preset distribution condition, the semantic mask corresponding to the object of interest is determined as the target mask. The semantic mask corresponding to the object of interest may be obtained by outputting the semantic segmentation model.

[0109] In some embodiments, the preset distribution condition may be that the range of depth values ​​in the depth information distribution is less than the range threshold. In some embodiments, the preset distribution condition may be that the standard deviation of depth values ​​in the depth information distribution is less than the standard deviation threshold. In some embodiments, the preset distribution condition may be that the variance of depth values ​​in the depth information distribution is less than the variance threshold. In some embodiments, the preset distribution condition may also be that multiple of the above conditions are satisfied at the same time. The relevant description of the preset distribution conditions is for illustrative purposes only and is not intended to limit the scope of this specification. Among them, the range threshold, standard deviation threshold, variance threshold may be a system default value, an empirical value, a manually preset value, etc. or any combination thereof, and may be set according to actual needs, and this specification does not limit this.

[0110] In some embodiments, the image region to which the object of interest belongs may be the region occupied by the object of interest in the image to be segmented. In some embodiments, the image region to which the object of interest belongs may be the region occupied by the object of interest in the image to be segmented and the region within a surrounding preset range. In some embodiments, the ratio of the number of pixels occupied by the surrounding preset range to the number of pixels occupied by the object of interest is less than a preset threshold. The preset threshold refers to the threshold corresponding to the pixel ratio, and the preset threshold can be set according to actual needs to ensure that the edge condition of the object of interest is reviewed in the surrounding preset range. Through this setting, a target mask with more accurate edge recognition can be obtained.

[0111] In some embodiments, the determination module 240 may obtain the first operation mask and the second operation mask through related operations, and further determine the target mask. For more information about the first operation mask and the second operation mask, see Figure 4 and its related description.

[0112] After acquiring the target mask, the processing device 110 (eg, an autonomous driving module) may perform downstream autonomous driving control based on the target mask.

[0113] In some embodiments, the processing device 110 may correct the position of the autonomous driving vehicle based on the target mask; and perform autonomous driving control based on the corrected position.

[0114] In some embodiments, the processing device 110 may determine the relative position (including distance, orientation, etc.) of the object of interest relative to the autonomous driving vehicle based on the target mask, determine the collision probability of the autonomous driving vehicle and the object of interest based on the relative position, and correct the position of the autonomous driving vehicle in response to the collision probability being greater than a probability threshold. For example, the processing device 110 may determine the collision probability based on one or more of the position of the autonomous driving vehicle, the relative position of the object of interest, the speed of the autonomous driving vehicle, the speed of the object of interest, etc.

[0115] There are many ways to correct the position of an autonomous driving vehicle. For example, the position of an autonomous driving vehicle can be corrected by adjusting the steering wheel, controlling the deceleration of the autonomous driving vehicle, etc., and there is no limitation here.

[0116] In some embodiments, the processing device 110 may correct the position of the autonomous driving vehicle based on the target mask so that the preset relative position between the autonomous driving vehicle and the object of interest satisfies the relative condition. The preset relative condition may be that the distance, orientation, etc. of the object of interest relative to the autonomous driving vehicle satisfies a specific condition. The preset relative condition may be in various forms, which may be set based on prior knowledge or actual needs, and is not limited here.

[0117] In some embodiments of the present specification, the position of the autonomous driving vehicle is corrected based on the target mask, so that the vehicle can be planned and controlled to execute a safe driving route, thereby improving the safety of autonomous driving.

[0118] In some embodiments of the present specification, by combining the semantic segmentation results with the point cloud depth information, the edge noise of the semantic elements can be filtered more accurately, and then the non-ground semantic elements can be accurately obtained, thereby assisting in achieving higher-precision positioning.

[0119] It should be noted that the above description of process 300 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 300 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0120] Figure 4 FIG. 1 is an exemplary schematic diagram of a target mask determination method according to some embodiments of the present specification. In some embodiments, the target mask determination method may be executed by the processing device 110 .

[0121] like Figure 4 As shown, in some embodiments, the processing device 110 can perform an AND operation based on the semantic mask 430 corresponding to the object of interest 410 and the point cloud mask 440 corresponding to the point cloud depth information 420 to obtain a first operation mask 450; dilate the first operation mask 450 to obtain a second operation mask 460; and perform a histogram analysis and voting operation on the second operation mask 460 to determine a target mask 470.

[0122] The semantic mask 430 refers to a mask of a region corresponding to the object of interest 410 on the image to be segmented.

[0123] In some embodiments, the processing device 110 may process the image to be segmented by using a trained semantic segmentation model to obtain a semantic mask 430 of the object of interest 410 and its corresponding region. The semantic mask may also be determined by other feasible methods, which are not limited here.

[0124] The point cloud mask 440 refers to a mask of a region corresponding to the object of interest 410 in the image to be segmented. The point cloud mask 440 may include depth value information of different pixels of the object of interest 410.

[0125] In some embodiments, the processing device 110 may project the point cloud depth information 420 onto the image to be segmented, divide the point cloud depth information 420 based on the region corresponding to the object of interest 410 in the image to be segmented, and obtain point cloud masks 440 corresponding to different objects of interest 410 .

[0126] The first operation mask 450 refers to a mask obtained by performing an AND operation on the semantic mask 430 and the point cloud mask 440 .

[0127] In some embodiments, the mask may be one of binary data, 4-ary data, decimal data, hexadecimal data, etc. AND operation refers to performing AND operation on the calculation data corresponding to the semantic mask 430 and the calculation data corresponding to the point cloud mask 440 .

[0128] In some embodiments, the processing device 110 may retain the pixels including both the semantic mask 430 and the point cloud mask 440 through an AND operation, and remove the pixels including only the semantic mask 430 or the point cloud mask 440, or neither of them, to obtain a first operation mask. The AND operation may retain the pixels having both the depth information and the semantic information.

[0129] The second operation mask 460 refers to a mask obtained by performing related operations on the first operation mask 450 .

[0130] In some embodiments, the processing device 110 may dilate the first operation mask 450 to determine the second operation mask 460. Dilation may expand the pixels of the image area to be dilated. Dilation may include, but is not limited to, horizontal dilation, vertical dilation, and omnidirectional dilation. For example, the processing device 110 may determine whether one or more pixel points (x, y) adjacent to the first operation mask 450 intersect with the pixel points corresponding to the first operation mask 450. In response to the existence of intersection, the pixel point (x, y) may be retained for dilation, and the mask of the retained pixel point (x, y) is the same as the mask corresponding to the pixel point with which it intersects, thereby determining the mask of the dilated area. Intersection may refer to the grayscale values ​​corresponding to the pixels being equal. The above description of dilation is for illustrative purposes only and is not intended to limit the scope of this specification.

[0131] In some embodiments, the second operation mask 460 refers to the sum of the first operation mask 450 and the mask of the dilation region.

[0132] In some embodiments, the second operation mask 460 may be obtained by performing other related operations on the first operation mask 450, which is not limited here.

[0133] It should be noted that in the semantic mask of the object of interest, the same object of interest may be divided into multiple independent regions, and the processing device 110 can fill the broken parts in the middle by dilation, connect the different regions of the same object of interest, and obtain a connected domain and its corresponding second operation mask. The connected domain can represent the corresponding region of a complete object of interest.

[0134] In some embodiments, before connecting domains are merged, grayscale conversion (eg, binarization) or other processing may be performed on the image to be segmented to better distinguish the connecting domains.

[0135] The histogram analysis can classify the depth values ​​of the pixels in the connected domain corresponding to the second operation mask 460 to obtain a grayscale histogram based on the depth values.

[0136] In some embodiments, the processing device 110 may determine the depth value with the highest probability or the largest number based on the grayscale histogram corresponding to the connected domain through a voting operation, and filter the pixel points corresponding to the remaining depth values ​​as noise points to determine the target mask 470 .

[0137] The above descriptions of the histogram analysis and voting operations are for illustrative purposes only and are not intended to limit the scope of this specification.

[0138] It should be noted that when only semantic segmentation is used to identify objects, due to insufficient sampling rate, fast LiDAR scanning speed, target size, long distance or occlusion, it is easy to cause low semantic recognition, non-target points are treated as targets, and target edge misdetection and other problems, which lead to inaccurate or invalid positioning. In some embodiments of the present specification, by performing AND operations, expansion, histogram analysis and voting operations on the semantic mask and the point cloud mask, the depth information contained in the point cloud mask can be combined on the basis of the semantic mask to assist in judging the edge of the target. Since there are more drastic and significant changes in depth values ​​at the edge of the target, combining the semantic mask with the point cloud mask can accurately filter out the edge noise of the semantic mask and obtain a refined target mask, which helps to improve the recognition accuracy of the object of interest and improve the accuracy of downstream positioning (for example, the positioning accuracy can be increased from several meters to centimeters).

[0139] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0140] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0141] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0142] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0143] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0144] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0145] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An image segmentation method, characterized in that: The method comprises: Acquire an image to be segmented, wherein the image to be segmented includes non-ground semantic elements; Determine an object of interest based on the image to be segmented and a trained semantic segmentation model, where the semantic segmentation model is a machine learning model; Based on the image to be segmented and the point cloud data, determining point cloud depth information corresponding to the image to be segmented; Based on the object of interest and the point cloud depth information, a target mask of the image to be segmented is determined.

2. The method according to claim 1, characterized in that The determining, based on the object of interest and the point cloud depth information, a target mask of the image to be segmented comprises: Performing an AND operation based on the semantic mask corresponding to the object of interest and the point cloud mask corresponding to the point cloud depth information to obtain a first operation mask; Expanding the first operation mask to obtain a second operation mask; A histogram analysis and a voting operation are performed on the second operation mask to determine the target mask.

3. The method according to claim 1, characterized in that The non-ground semantic elements include at least one or more of traffic signs, traffic lights and fences.

4. The method according to claim 1, characterized in that The method further comprises: Correcting the position of the autonomous driving vehicle based on the target mask; Automatic driving control is performed based on the corrected position.

5. The method according to claim 1, characterized in that Determining the point cloud depth information of the image to be segmented includes: In response to a ratio of a first quantity of point cloud depth information to a second quantity of image pixel point information satisfying a preset condition, multiple frames of the point cloud data are fused to obtain a third quantity of point cloud depth information, where the third quantity is determined based on the number of the objects of interest.

6. The method according to claim 1, characterized in that The semantic segmentation model is obtained in the following way: Get a sample image; Annotating the sample image based on the mask category of the non-ground semantic element corresponding to each pixel in the sample image; The labeled sample images are input into the initial semantic segmentation model for training to determine the semantic segmentation model.

7. The method according to claim 1, characterized in that The step of determining the point cloud depth information of the image to be segmented based on the image to be segmented and the point cloud data includes: Based on the coordinate conversion relationship between the image coordinate system and the laser radar coordinate system, the point cloud data is mapped to the image to be segmented, and the depth information of the point cloud is determined.

8. An image segmentation system, characterized in that: The system comprises: An acquisition module, used for acquiring an image to be segmented, wherein the image to be segmented includes non-ground semantic elements; A segmentation module, used to determine an object of interest based on the image to be segmented and a trained semantic segmentation model, wherein the semantic segmentation model is a machine learning model; An analysis module, configured to determine point cloud depth information of the image to be segmented based on the image to be segmented and the point cloud data; A determination module is used to determine a target mask of the image to be segmented based on the object of interest and the point cloud depth information.

9. An image segmentation device, characterized in that: The apparatus comprises at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is used to execute at least part of the computer instructions to implement the image segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the image segmentation method according to any one of claims 1 to 7.

Images