Target segmentation method, video panoramic segmentation method and device

Abstract

The embodiment of the invention provides a target segmentation method and device, and a video panoramic segmentation method and device. Comprising the following steps: encoding an image to obtain an encoded image; performing semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image; and based on the prompt feature of the target object, the encoded image and the semantic feature of the encoded image, processing is carried out to obtain a mask of the target object, and the multi-scale prompt feature of the target object is obtained according to the prompt information of the target object. By adopting the means, the prediction result can be perfected by combining the prompt characteristics of the target object based on the semantic mask decoding processing, so that the prediction is more accurate, and the contour is more fit; meanwhile, universality and precision are guaranteed, and redundancy of over-segmentation can be made up.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a target segmentation method, a video panoramic segmentation method and a device. Background Art

[0002] Segmentation tasks (including semantic segmentation, instance segmentation, and panoptic segmentation) are a basic research direction in the field of computer vision, which aims to perceive images at the pixel level. In recent years, data-driven semantic segmentation algorithms have also made great progress and are widely used in many fields such as autonomous driving, smart cities, and medical image processing. Recently, with the development of large natural language models and large visual models, large model perception tasks based on general scenarios have gradually become the key to the continuous optimization of basic autonomous driving algorithms.

[0003] The current supervised training segmentation task requires large-scale data and detailed labels, and manual labeling consumes a lot of manpower and financial resources. The problem of pixel-level segmentation is particularly prominent. In actual scenarios, limited labeled samples will cause overfitting of deep learning network training, resulting in weak generalization ability, inability to effectively distinguish difficult samples, and ultimately leading to wrong predictions, which is not conducive to commercial applications.

[0004] Currently, when performing target segmentation, a picture is given and a prompt is used to specify the area. For example, the prompt can be a mask, point, box, or text, and finally the mask prediction result of the corresponding area is output. This method has a simple network design and strong versatility. However, this method cannot cover all areas in the picture. For specific objects, the segmentation accuracy is obviously insufficient and the edges are not close enough. Summary of the invention

[0005] The present application discloses a target segmentation method, a video panoramic segmentation method and a device, which can effectively improve the segmentation accuracy.

[0006] In a first aspect, an embodiment of the present application provides a target segmentation method, comprising:

[0007] Acquire an image, and perform encoding processing on the image to obtain an encoded image;

[0008] Performing semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image;

[0009] The mask of the target object is obtained by processing based on the prompt feature of the target object in the image, the encoded image, and the semantic feature of the encoded image, wherein the prompt feature of the target object is obtained based on the prompt information of the target object.

[0010] In the embodiment of the present application, the semantic features of the encoded image are obtained by performing semantic mask decoding processing on the encoded image, and then the mask of the target object is obtained by performing processing based on the prompt features of the target object, the encoded image, and the semantic features of the encoded image. By adopting this method, the prediction result can be improved based on the semantic mask decoding processing and combined with the prompt features of the target object, making the prediction more accurate and the contour more fitting; at the same time, the versatility and accuracy are guaranteed, and the redundancy of over-segmentation can be compensated.

[0011] In a possible implementation, the processing based on the prompt feature of the target object, the encoded image, and the semantic feature of the encoded image to obtain the mask of the target object includes:

[0012] Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object;

[0013] Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object;

[0014] Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

[0015] By adopting this method, the segmentation contour can be made more consistent and accurate.

[0016] In a possible implementation, the method further includes:

[0017] Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object;

[0018] Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images;

[0019] Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively;

[0020] The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

[0021] The multi-scale hint features obtained based on the multi-scale hint processing of this example can further help improve the segmentation accuracy.

[0022] In a possible implementation, the processing of the encoded scattered features corresponding to the images of different scales to obtain the prompt features of the target object includes:

[0023] Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features;

[0024] Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

[0025] In a possible implementation, the semantic mask of the encoded image, the category prediction result of the target object in the encoded image, and the semantic features of the encoded image are obtained by inputting the encoded image into a semantic mask decoding module for processing, wherein the semantic mask decoding module includes a first transformer structure, an upsampling layer, and a feedforward neural network.

[0026] In one possible implementation, the mask of the target object is obtained by inputting the multi-scale cue features of the target object, the encoded image, and the semantic features of the encoded image into a category-free mask decoding module for processing, wherein the category-free mask decoding module includes a second transformer structure, a third transformer structure, a token-image feature transition attention layer, an upsampling layer, and a feedforward neural network.

[0027] In a possible implementation, the prompt feature of the target object is obtained by inputting the semantic mask, the category prediction result of the target object and the prompt information of the target object into a multi-scale prompt encoding module for processing, and the multi-scale prompt encoding module includes a progressive refinement prompt unit, a prompt encoding unit and a multi-scale prompt adapter.

[0028] In a second aspect, an embodiment of the present application provides a target segmentation network, the network comprising an encoding module, a semantic mask decoding module and a category-free mask decoding module, wherein:

[0029] The encoding module is used to encode the acquired image to obtain an encoded image;

[0030] The semantic mask decoding module is used to perform semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image;

[0031] The category-free mask decoding module is used to obtain a mask of the target object by processing based on the prompt features of the target object in the image, the encoded image, and the semantic features of the encoded image, wherein the prompt features of the target object are obtained based on the prompt information of the target object.

[0032] In a possible implementation, the classless mask decoding module is used to:

[0033] Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object;

[0034] Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object;

[0035] Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

[0036] In a possible implementation, the network further includes a multi-scale hint encoding module, configured to:

[0037] Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object;

[0038] Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images;

[0039] Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively;

[0040] The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

[0041] In a possible implementation, the multi-scale hint encoding module is further used to:

[0042] Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features;

[0043] Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

[0044] In a third aspect, an embodiment of the present application provides a video panorama segmentation method, including:

[0045] Acquire a video frame, and perform initial segmentation processing on the video frame to obtain a first foreground image and a first background image in the video frame;

[0046] Processing the video frame, the first foreground image, and the multi-scale prompt features of the first foreground image through a first target segmentation network to obtain a second foreground image, wherein the segmentation accuracy corresponding to the second foreground image is greater than that of the first foreground image, and the second foreground image corresponds to the first foreground image;

[0047] Processing the first background image and the prompt features of the first background image through a second object segmentation network to obtain a reference mask of the first background image;

[0048] performing feature fusion processing based on the reference mask of the first background image, the first background image, and the second foreground image to obtain a second background image, wherein the segmentation accuracy corresponding to the second background image is greater than that of the first background image, and the second background image corresponds to the first background image;

[0049] The first target segmentation network and / or the second target segmentation network is a network as described in any one of claims 8 to 11.

[0050] Based on the above processing, this example can achieve panoramic segmentation of both countable objects and uncountable objects in the image with high segmentation accuracy.

[0051] In a possible implementation, the method further includes:

[0052] Based on the second foreground image, the second background image, the third foreground image, and the third background image, temporal feature matching processing is performed to obtain a fourth foreground image and a fourth background image, wherein the third foreground image and the third background image are both obtained by processing the video frame through a third target segmentation network, the second foreground image, the third foreground image, and the fourth foreground image correspond to each other, the second background image, the third background image, and the fourth background image correspond to each other, the segmentation accuracy of the fourth foreground image is greater than that of the second foreground image and the third foreground image, and the segmentation accuracy of the fourth background image is greater than that of the second background image and the third background image.

[0053] This embodiment optimizes things and stuff using multiple general segmentation frameworks based on the existing panoramic segmentation framework, and optimizes multi-target tracking in combination with a temporal feature matching module. The modules work together to improve the effect of panoramic segmentation.

[0054] In a fourth aspect, an embodiment of the present application provides a target segmentation device, including:

[0055] The encoding module is used to acquire an image and perform encoding processing on the image to obtain an encoded image;

[0056] A processing module, used for performing semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image;

[0057] The segmentation module is used to obtain a mask of the target object by processing based on the prompt features of the target object in the image, the encoded image, and the semantic features of the encoded image, wherein the prompt features of the target object are obtained based on the prompt information of the target object.

[0058] In a possible implementation, the segmentation module is used to:

[0059] Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object;

[0060] Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object;

[0061] Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

[0062] In a possible implementation, a prompt module is also included, which is further used to:

[0063] Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object;

[0064] Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images;

[0065] Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively;

[0066] The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

[0067] In a possible implementation, the prompt module is further used to:

[0068] Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features;

[0069] Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

[0070] In a possible implementation, the processing module is a semantic mask decoding module, wherein the semantic mask decoding module includes a first transformer structure, an upsampling layer and a feedforward neural network.

[0071] In a possible implementation, the segmentation module is a category-free mask decoding module, wherein the category-free mask decoding module includes a second transformer structure, a third transformer structure, a token-image feature transition attention layer, an upsampling layer and a feedforward neural network.

[0072] In a possible implementation, the prompt module is a multi-scale prompt encoding module, and the multi-scale prompt encoding module includes a progressive refinement prompt unit, a prompt encoding unit and a multi-scale prompt adapter.

[0073] In a fifth aspect, an embodiment of the present application provides a video panorama segmentation device, including:

[0074] A first processing module, used for acquiring a video frame, and performing initial segmentation processing on the video frame to obtain a first foreground image and a first background image in the video frame;

[0075] A second processing module is used to process the video frame, the first foreground image and the multi-scale prompt feature of the first foreground image through a first target segmentation network to obtain a second foreground image, the segmentation accuracy corresponding to the second foreground image is greater than that of the first foreground image, and the second foreground image corresponds to the first foreground image;

[0076] a third processing module, configured to process the first background image and the prompt feature of the first background image through a second object segmentation network to obtain a reference mask of the first background image;

[0077] a fourth processing module, configured to perform feature fusion processing based on a reference mask of the first background image, the first background image, and the second foreground image to obtain a second background image, wherein the segmentation accuracy corresponding to the second background image is greater than that of the first background image, and the second background image corresponds to the first background image;

[0078] The first target segmentation network and / or the second target segmentation network is a network as described in any one of claims 8 to 11.

[0079] In a possible implementation manner, the apparatus further includes a fifth processing module, configured to:

[0080] Based on the second foreground image, the second background image, the third foreground image, and the third background image, temporal feature matching processing is performed to obtain a fourth foreground image and a fourth background image, wherein the third foreground image and the third background image are both obtained by processing the video frame through a third target segmentation network, the second foreground image, the third foreground image, and the fourth foreground image correspond to each other, the second background image, the third background image, and the fourth background image correspond to each other, the segmentation accuracy of the fourth foreground image is greater than that of the second foreground image and the third foreground image, and the segmentation accuracy of the fourth background image is greater than that of the second background image and the third background image.

[0081] In a sixth aspect, an embodiment of the present application provides a target segmentation device, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute a method provided in any possible implementation manner of the first aspect.

[0082] In the seventh aspect, an embodiment of the present application provides a video panoramic segmentation device, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute a method provided in any possible implementation manner of the third aspect.

[0083] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement a method provided in any possible implementation manner of the first aspect and / or any possible implementation manner of the third aspect.

[0084] In a ninth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes a method provided in any possible implementation of the first aspect and / or any possible implementation of the third aspect.

[0085] In a tenth aspect, an embodiment of the present application provides a vehicle, comprising a device provided by any possible implementation of the fourth aspect and / or any possible implementation of the fifth aspect.

[0086] It can be understood that the network described in the second aspect, the device described in the fourth aspect, the device described in the fifth aspect, the device described in the sixth aspect, the device described in the seventh aspect, the computer storage medium described in the eighth aspect, the computer program product described in the ninth aspect, or the vehicle described in the tenth aspect are all used to execute any method provided in the first aspect and any method provided in the third aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The following is an introduction to the drawings used in the embodiments of the present application.

[0088] Figure 1 is a schematic diagram of a target segmentation system in an autonomous driving scenario provided by an embodiment of the present application;

[0089] Figure 2 It is a flowchart of a target segmentation method provided in an embodiment of the present application;

[0090] Figure 3 It is a schematic diagram of the structure of a target segmentation network provided in an embodiment of the present application;

[0091] Figure 4 It is a structural diagram of a multi-head asynchronous fusion mask decoding module provided in an embodiment of the present application;

[0092] Figure 5 It is a schematic diagram of a model training method of a multi-head asynchronous fusion mask decoding module provided in an embodiment of the present application;

[0093] Figure 6 is a schematic diagram of a multi-scale prompt encoding module provided in an embodiment of the present application;

[0094] Figure 7 It is a schematic diagram of a progressively refined prompt unit processing provided in an embodiment of the present application;

[0095] Figure 8 is a schematic diagram of a multi-scale prompt adapter processing provided by an embodiment of the present application;

[0096] Fig. 9 It is a flowchart of a video panorama segmentation method provided in an embodiment of the present application;

[0097] Fig.10 This is a schematic diagram of a video panorama segmentation process provided by an embodiment of the present application;

[0098] Fig.11 is a structural schematic diagram of a target segmentation device provided in an embodiment of the present application;

[0099] Fig.12 It is a structural schematic diagram of a video panorama segmentation device provided in an embodiment of the present application;

[0100] Fig.13 is a schematic structural diagram of another target segmentation device provided in an embodiment of the present application;

[0101] Fig.14 It is a structural schematic diagram of another video panoramic segmentation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0103] To facilitate understanding, the following first introduces relevant terms that may be involved in the embodiments of the present application.

[0104] (1) Autonomous driving

[0105] Autonomous driving, also known as intelligent driving or assisted driving, is an important direction for the development of vehicle intelligence. With the development of perception technology and the improvement of chip capabilities, intelligent driving provides people with more and more rich driving functions and gradually realizes different levels of driving experience. The Society of Automotive Engineers (SAE) provides a driving automation classification standard, including driving levels L0 to L5, where L0 is no automation and the human driver has full control over the vehicle. During driving, the driver can get warnings or assistance from the driving system, such as autonomous emergency braking (AEB), blind spot monitoring (BSM) or lane departure warning (LDW). Level 1 is driving assistance, and the driving operation is completed by the human driver and the driving system. The driving system can provide driving assistance for the steering wheel or acceleration and deceleration operations through the driving environment, and other driving operations are performed by the human driver, such as adaptive cruise control (ACC) or lane keep assistance / support (LKA / LKS). Level 2 is partial automation, which provides driving assistance for the steering wheel and multiple acceleration and deceleration operations through the driving environment, and other driving actions are performed by the human driver, such as a combination of adaptive cruise control (ACC) and lane keep assistance (LKA / LKS). Level 3 is conditional automation, where the driving system can complete all driving operations, but the human driver needs to respond to the request of the driving system at the appropriate time, that is, the human driver needs to be prepared to take over the driving system; Level 4 is highly automated, where the driving system can complete all driving operations, and the human driver does not necessarily need to respond to the request of the driving system. For example, when the road and environmental conditions permit (such as closed parks, highways, urban roads or fixed driving routes, etc.), the human driver may not take over the driving; Level 5 is fully automated, where the driving system can autonomously complete driving operations under various road and environmental conditions that human drivers can cope with. It can be seen that at levels L0 to L2, the driving system mainly provides support for the driver, and the driver still needs to supervise the driving and steer, brake or accelerate as needed to ensure safety. At levels L3 to L5, the driving system can complete all driving operations on behalf of the driver. At level L3, the driver must be prepared to take over the driving. At levels L4 and L5, the driving system can achieve full driving under some conditions and all conditions, and the driver can choose whether to take over.

[0106] The above classification is an example. With the evolution of technology or different regulations in different countries or regions, the above classification may change. For example, the vehicle automation classification proposed by the Ministry of Industry and Information Technology of China includes 6 levels of vehicle driving automation, among which Levels 0-2 are driving assistance, the system assists humans in performing dynamic driving tasks, and the driving subject is still the driver; Levels 3-5 are automatic driving, the system replaces humans in performing dynamic driving tasks under the designed operating conditions, and when the function is activated, the driving subject is the system. The names and definitions of each level are as follows: Level 0 driving automation (emergency assistance) The system cannot continuously perform the lateral or longitudinal motion control of the vehicle in dynamic driving tasks, but has the ability to continuously perform partial target and event detection and response in dynamic driving tasks. Level 1 driving automation (partial driver assistance) The system continuously performs the lateral or longitudinal motion control of the vehicle in dynamic driving tasks under its designed operating conditions (or called the design operating range ODD), and has the ability to detect and respond to partial targets and events that are compatible with the lateral or longitudinal motion control of the vehicle being performed. Level 2 driving automation (combined driver assistance) systems continuously perform lateral and longitudinal motion control of the vehicle in dynamic driving tasks under their designed operating conditions, and have the ability to detect and respond to some targets and events that are compatible with the lateral and longitudinal motion control of the vehicle being performed. Level 3 driving automation (conditionally automated driving) systems continuously perform all dynamic driving tasks under their designed operating conditions. Level 4 driving automation (highly automated driving) systems continuously perform all dynamic driving tasks under their designed operating conditions and automatically implement minimum risk strategies. Level 5 driving automation (fully automated driving) systems continuously perform all dynamic driving tasks under any drivable conditions and automatically implement minimum risk strategies. Among them, lateral control is mainly used for vehicle steering control, for example, controlling the steering wheel torque or angle to control the direction of the vehicle; longitudinal control is mainly used for vehicle speed control, for example, controlling the brake pedal, accelerator pedal, or gear position to control the acceleration / deceleration, braking, etc. of the vehicle.

[0107] Regardless of the classification method adopted, the description of the embodiments of the present application can be applied to the above-mentioned automatic driving systems that require partial or full participation in vehicle driving.

[0108] (2) Semantic Segmentation: Analyze objects in an image or a video stream and mark their categories pixel by pixel.

[0109] (3) Instance Segmentation: Instance segmentation is to segment objects in an image or video at the pixel level.

[0110] (4) Panoptic Segmentation: A combination of semantic segmentation and instance segmentation, which not only identifies the pixel-level categories of the entire scene, but also segments individual objects on an individual basis.

[0111] (5) Stuff: refers to the uncountable objects in panoramic segmentation, mainly objects such as the sky, road surface, and buildings that appear in large areas during segmentation. In panoramic segmentation, only the corresponding areas need to be identified. It can refer to the background in the image.

[0112] (6) Things: refers to countable objects in panoptic segmentation, mainly objects with individual entities such as people, cars, and bicycles. Individual entities must be distinguished in panoptic segmentation. It can refer to the foreground in the image.

[0113] The above exemplary description of the concepts can be applied in the following embodiments.

[0114] In application scenarios such as autonomous driving, pixel-level accurate classification and perception are required under various road conditions. In addition, there are some uncommon objects during driving, such as special-shaped vehicles and unknown objects, which are very important for driving safety. Therefore, it is necessary to provide a method with higher segmentation accuracy to achieve panoramic segmentation while ensuring versatility and accuracy.

[0115] The following will describe the system architecture of the embodiment of the present application in detail with reference to the accompanying drawings. Figure 1 , Figure 1 1 is a schematic diagram of an object segmentation system in an autonomous driving scenario applicable to an embodiment of the present application, the system comprising a vehicle 101 and a server 102. Among them:

[0116] Here, vehicle 101 refers to a vehicle equipped with an automatic driving system. The automatic driving system is not limited to a fully automatic driving system, a highly automatic driving system, a conditional automatic driving system, or a partially automatic driving system, and those skilled in the art can understand that non-fully manual driving systems that provide intelligent driving can be covered under this concept.

[0117] Exemplarily, depending on the power source of the vehicle, the vehicle may be a new energy vehicle or a traditional vehicle, etc., wherein a traditional vehicle refers to a fuel vehicle, such as a gasoline vehicle, a diesel vehicle, etc., and a new energy vehicle may be, for example, an electric vehicle (EV), a hybrid electric vehicle (HEV), an extended-range electric vehicle (range extended EV), a plug-in hybrid vehicle (Plug-in HEV), a fuel cell vehicle or other new energy vehicles, without specific limitation herein.

[0118] Among them, cameras and radars are deployed on the vehicle, wherein the camera is used to collect image data of the vehicle's current surroundings, and the radar is used to collect point cloud data of the vehicle's current surroundings. The radar includes at least one of a laser radar Lidar, a millimeter-wave radar Radar, and the like. Based on the installation position of the camera on the vehicle, the camera can be divided into, for example, a front-view camera, a surround-view camera, a rear-view camera, and a side-view camera, and the like; based on the structural division of the camera, the camera can be divided into, for example, a monocular camera, a binocular camera, a wide-angle camera, and the like. Here, the embodiment of the present application does not limit the number of cameras configured for the vehicle. For safety reasons, the camera on the vehicle needs to be able to collect 360-degree image data around the vehicle body.

[0119] Here, the server 102 is a device with computing capabilities. The server 102 can be, for example, a server deployed on the network side (e.g., a server for intelligent driving processing), or a component or chip in the server. In some possible embodiments, the server 102 can also be a system-level device or a computing device cluster composed of multiple servers. The server 102 can be deployed in a cloud environment or an edge environment, which is not specifically limited in the embodiments of the present application.

[0120] Above Figure 1 The figure shows an implementation method of the embodiment of the present application. In this implementation method, the target segmentation method provided by the embodiment of the present application can be implemented on the server 102.

[0121] In another possible implementation, the embodiment of the present application can also be used in a vehicle, for example, it can be an onboard device (onboard unit, OBU) in a vehicle with a target segmentation function, or it can be set in a sensor with a target segmentation acquisition and processing function. Among them, the onboard device can include but is not limited to a vehicle terminal, a vehicle controller, a vehicle module, a vehicle module, a vehicle component, a vehicle chip, a vehicle unit, a vehicle radar, an electronic control unit (electronic control unit, ECU), a domain controller (domain controller, DC) and other devices.

[0122] The embodiments of the present application can be applied to the field of autonomous driving vehicles, and can also be used in medical imaging, scene understanding and other scenes that require segmentation, etc. This solution does not limit this.

[0123] The architecture of the embodiment of the present application is described above, and the method of the embodiment of the present application is described in detail below.

[0124] Reference Figure 2 FIG. 1 is a flow chart of a target segmentation method provided in an embodiment of the present application. Optionally, the method can be applied to the aforementioned target segmentation system, for example Figure 1 The target segmentation system shown in Figure 2 The target segmentation method shown may include steps 201-203. It should be understood that for the convenience of description, this application is described in the order of 201-203, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. The following description takes the execution subject of steps 201-203 of the target segmentation method as a server as an example, and this application is also applicable to other execution subjects. Steps 201-203 are as follows:

[0125] 201. Acquire an image, and perform encoding processing on the image to obtain an encoded image.

[0126] For example, the server obtains the images collected by the vehicle from the vehicle in real time. Of course, it is also possible not to obtain in real time, and this solution does not limit this.

[0127] In a possible implementation, the acquired image is input into an encoder for processing to obtain an encoded image.

[0128] 202. Perform semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image.

[0129] A semantic mask can be understood as a binary image that indicates the area that semantic segmentation should focus on.

[0130] In a possible implementation, the semantic features of the encoded image can be obtained by inputting the encoded image into a preset neural network for processing. For example, the preset neural network can be as follows: Figure 3 The semantic mask decoding module in the target segmentation network shown above is only an example and this solution does not limit this.

[0131] 203. Perform processing based on a prompt feature of the target object in the image, the encoded image, and a semantic feature of the encoded image to obtain a mask of the target object, wherein the prompt feature of the target object is obtained based on the prompt information of the target object.

[0132] The prompt information of the target object may be, for example, at least one of prompt information such as points, a detection box, text, and a mask of the target object.

[0133] In a possible implementation, the prompt feature of the target object can be obtained in the following way:

[0134] By processing the images of different scales containing the target object respectively, the masks corresponding to the images of different scales are obtained. Wherein, the images of different scales containing the target object include prompt information of the target object (for example, the prompt information can be multi-scale prompt information). Then, based on the masks corresponding to the images of different scales, the semantic mask of the encoded image and the category prediction result of the target object, the scattered sampling features corresponding to the images of different scales are obtained. Wherein, the semantic mask of the encoded image and the category prediction result of the target object are obtained by processing the encoded image. The category prediction result of the target object indicates the category corresponding to the target object. In a possible implementation, by inputting the encoded image into a preset neural network for processing, the semantic mask of the encoded image, the category prediction result of the target object in the encoded image and the semantic feature of the encoded image can be obtained. Specifically, by inputting the encoded image into the semantic mask decoding module for processing, an intermediate feature (i.e., the semantic feature of the encoded image) can be obtained, and the semantic mask decoding module continues to process the semantic feature of the encoded image to obtain the semantic mask of the encoded image and the category prediction result of the target object in the encoded image. Then, by encoding the scattered sampling features corresponding to the images of different scales, the encoded scattered features corresponding to the images of different scales are obtained. Then, the encoded scattered features corresponding to the images of different scales are processed to obtain the prompt features of the target object. For example, based on the encoded scattered features corresponding to the small-scale images in the images of different scales, the small-scale features are obtained; and then based on the small-scale features and the encoded scattered features corresponding to the large-scale images in the images of different scales, the prompt features of the target object are obtained (for example, the prompt features are multi-scale prompt features).

[0135] Exemplarily, the hint feature of the target object is to input the semantic mask of the encoded image, the category prediction result of the target object and the hint information of the target object into the following Figure 3 The multi-scale hint encoding module shown is used to process the image.

[0136] In a possible implementation, processing is performed based on the prompt features of the target object and the semantic features of the encoded image to obtain the abstract contour features of the target object. Then, processing is performed based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain the spatial detail features of the target object. Further, processing is performed based on the detail features of the target object and the contour features of the target object to obtain the mask of the target object.

[0137] Exemplarily, the mask of the target object can be obtained by inputting the prompt feature of the target object, the encoded image, and the semantic feature of the encoded image into a preset neural network for processing. Exemplarily, the preset neural network can be, for example, as follows: Figure 3 The agnostic mask decoding module shown.

[0138] In a possible implementation, steps 202-203 may be replaced by:

[0139] Performing semantic mask decoding processing on the encoded image to obtain a semantic mask of the encoded image and a category prediction result of a target object in the encoded image;

[0140] The mask of the target object is obtained by processing the prompt features of the target object and the encoded image, wherein the prompt features of the target object are obtained by processing the semantic mask of the encoded image, the category prediction result of the target object and the prompt information of the target object.

[0141] For the introduction of this part, please refer to the record of the aforementioned step 203, which will not be repeated here.

[0142] In the embodiment of the present application, the semantic features of the encoded image are obtained by performing semantic mask decoding processing on the encoded image, and then the mask of the target object is obtained by performing processing based on the prompt features of the target object, the encoded image, and the semantic features of the encoded image. By adopting this method, the prediction result can be improved based on the semantic mask decoding processing and combined with the prompt features of the target object, making the prediction more accurate and the contour more fitting; at the same time, the versatility and accuracy are guaranteed, and the redundancy of over-segmentation can be compensated.

[0143] Reference Figure 3As shown, it is a schematic diagram of a target segmentation network provided by an embodiment of the present application. The target segmentation network includes an image encoding module, a multi-head asynchronous fusion mask decoding module and a multi-scale prompt prompt encoding module. Among them, the multi-head asynchronous fusion mask decoding module includes a semantic mask decoding module and an agnostic mask decoding module. Exemplarily, after the input image is processed by the image encoding module, an encoded image is obtained. By inputting the encoded image into the semantic mask decoding module and the agnostic mask decoding module of the multi-head asynchronous fusion mask decoding module for processing, the valid mask validmask of the target object is finally obtained. Among them, the input of the agnostic mask decoding module also includes the semantic features of the encoded image obtained by processing the semantic mask decoding module, and the multi-scale prompt features of the target object. The multi-scale prompt feature of the target object is obtained by processing the multi-scale prompt information corresponding to the input target object (such as points, detection boxes, texts, masks, etc.) by the multi-scale prompt encoding module, the semantic mask obtained by the semantic mask decoding module, and the category prediction result of the target object.

[0144] The following introduces the structures of the semantic mask decoding module and the agnostic mask decoding module in the multi-head asynchronous fusion mask decoding module.

[0145] In one possible implementation, referring to Figure 4 As shown, the above semantic mask decoding module may include a self-attention layer, a residual connection and a normalized add&norm layer, a feedforward neural network FFN layer, an upsample layer, etc. The input of the semantic mask decoding module may include image embedding features (i.e., the encoded image) and semantic query features (i.e., the queries in the input attention mechanism). The self-attention layer, the residual connection and the normalized add&norm layer, and the feedforward neural network FFN layer constitute a transformer unit structure, and the transformer structure can be cascaded, such as Figure 4The ×3 on the left side of the figure represents a cascade of three transformer layers. After the image embedding features and semantic query features are processed through three cascaded transformer layers, the last transformer layer outputs the semantic features of the encoded image. The semantic features of the encoded image are processed through upsampling and feedforward neural network, and the features are multiplied with the image embedding features to obtain the final output semantic mask. At the same time, the semantic features of the encoded image output by the last transformer layer are processed through feedforward neural network to obtain the category prediction result of the target object.

[0146] Reference Figure 4 As shown, the agnostic mask decoding module may include a self-attention layer, a residual connection and a normalized add&norm layer, a feedforward neural network FFN layer, an upsample layer, a token-to-image feature transition attention layer, etc. The input of the agnostic mask decoding module includes image embedding features, semantic query features, prompt tokens, agnostic output queries, and semantic features of the encoded image. The self-attention layer, the residual connection and the normalized add&norm layer, the feedforward neural network FFN layer, the upsample layer, and the token-to-image feature transition attention layer form a two-way transformer unit, and this two-way transformer structure can be cascaded, such as Figure 4 The ×3 on the right side of the figure represents a cascade of three layers of dual-path transformers. Figure 4The left network structure in the agnostic mask decoding module is similar to the semantic mask decoding module. The input is also image embedding features and semantic query features, while the input of the right network structure is prompt tokens and agnostic output queries. After the relevant inputs are simultaneously passed through three layers of cascaded dual-path transformers, the features obtained by upsampling and feedforward network output of the last layer of transformer are multiplied with the image embedding features to obtain the final output of the category-free mask (i.e., effective mask). At the same time, the features obtained by the last layer of transformer output through the feedforward neural network are obtained to obtain the Intersection over Union (IOU) confidence. There is a two-way information interaction between the left network structure and the right network structure. First, the image embedding features on the left are introduced into the right transformer through the token-image feature transition attention layer to introduce semantic features. Then, the transformer output of the right transformer is introduced into the self-attention network of the left transformer to introduce category-free features. The network structure on the right can obtain the abstract contour features of the target object, and the network structure on the left can obtain the spatial detail features of the target object based on the abstract contour features. The mask of the target object is obtained by processing the abstract contour features and the spatial detail features of the target object.

[0147] It should be noted that Figure 4 The network architecture of the multi-head asynchronous fusion mask decoding module shown is only an example, and it can also be other structures, which is not limited in this solution.

[0148] like Figure 5As shown, a model training method for a multi-head asynchronous fusion mask decoding module provided by an embodiment of the present application. Among them, the pre-training adopts an asynchronous iterative training scheme. During the training, it is necessary to first train the semantic mask decoding module and then train the category-free agnostic mask decoding module. In order to achieve the fusion effect, for example, the semantic mask decoding module is first trained with data from an existing autonomous driving scene, and the parameters of the agnostic mask decoding module remain unchanged during the process. The parameters of the semantic decoding are adjusted based on the semantic mask prediction results output by the semantic mask decoding module and the real data of the existing autonomous driving scene until the training is completed. Then, the category-free mask decoding module is trained with a category-free segmentation data set. The training data can be an existing category-free segmentation data set (such as SA-1B), or the categories in the categorized segmentation data set (such as ADE20K, CityScapes) are removed and used as category-free data. When training the category-free mask decoding module, the parameters of the semantic mask decoding module remain unchanged, and the features output by it are introduced into the category-free mask decoding module through an asynchronous fusion structure. When the agnostic mask decoding module is trained, the above-mentioned autonomous driving data can be used for fine-tuning. For example, after the network is loaded with the pre-training parameters of the above process, the segmentation data of the autonomous driving is processed without categories, and fine-tuned according to the above process. Based on the above training, a trained multi-head asynchronous fusion mask decoding module can be obtained.

[0149] The multi-scale prompt encoding module is introduced below.

[0150] Due to the multi-scale characteristics of mask decoding, the hint encoding is extended to multiple scales to further improve the accuracy of segmentation. For example, for instance-level targets, such as moving objects (cars, bicycles, and pedestrians), and static indicator targets, such as signboards and traffic lights, the existing bounding box (Bounding Box, bbox) corresponding to the target and the output of semantic mask decoding can be used together to obtain instance-level hints. At the same time, the segmentation accuracy is further improved based on the refined processing of hints.

[0151] For example, refer to Figure 6 , which is a schematic diagram of a multi-scale prompt encoding module provided in an embodiment of the present application. The multi-scale prompt encoding module may include a progressively refined prompt unit, a prompt encoding unit, and a multi-scale prompt adapter.

[0152] The following is an introduction to the progressively refined prompt unit of the scale prompt encoding module. Among them, for the progressively refined prompt unit, three states can be used for prompting, for example, the three states of "definitely yes", "definitely no" and "fuzzy" can be included, and they can be improved step by step. Figure 7 As shown, the images of different scales can be segmented to obtain initialization masks of different scales. After the confidence and corrosion operations are performed on the initialization masks of different scales, the three-state graphs corresponding to the images of different scales are obtained. For example, different color areas are set to correspond to the above three different states, such as the red area (not shown in the figure) is the "definitely yes" category, the green area (not shown in the figure) is the "definitely no" category, and the yellow area (not shown in the figure) is the "fuzzy" category. Then, the three-state graphs of each scale are scattered sampled (excluding the singular value area), for example, the same absolute sampling rate is used for sampling (such as sampling with an interval of 5 pixels), and multi-scale scattered sampling features can be obtained.

[0153] Prompt encoding processing is performed based on the multi-scale scattered sampling features to obtain the encoded scattered features corresponding to the multi-scale images.

[0154] For the multi-scale prompt adapter, the segmentation accuracy is further improved by extending the encoded scattered features output by the prompt encoding unit to multiple scales. Figure 8 As shown in the figure, the encoded scattered features corresponding to the large-scale and small-scale images are respectively input into the multi-scale prompt adapter for processing. The small-scale three-state graph used for prompt (i.e., the encoded scattered feature graph corresponding to the small-scale image) is processed by the multi-layer perception network and the self-attention network to obtain the small-scale features. The large-scale three-state graph (i.e., the encoded scattered feature graph corresponding to the large-scale image) is down-sampled and then the features output by the multi-layer perception network are fused with the above-mentioned small-scale features. The fused features are up-sampled and fused with the features of the large-scale three-state graph to finally obtain the multi-scale prompt features of the target object.

[0155] In this embodiment, the actual scene image input can be encoded by a unified backbone network, and the output features are processed by a multi-head asynchronous fusion mask decoding module, and prompted by a multi-scale prompt, which can make the target prediction more accurate and the contour more fitting. This solution can make full use of reasonable prompts for highly generalized category expansion and pixel-level segmentation prediction, without over-reliance on the annotation of existing data, which greatly reduces the annotation cost.

[0156] On the other hand, compared with the Segment Anything Model (SAM), this solution optimizes decoding in a targeted manner, such as multi-scale encoding structure and pre-trained semantic branches, which can further improve the accuracy of segmentation. This solution can solve the shortcomings of existing general segmentation models and the limitations of practical application scenarios, and improve the functions of SAM on a functional basis.

[0157] The above embodiment is introduced by taking the object segmentation method as an example. The following is an illustration of applying the object segmentation method to video panorama segmentation.

[0158] Reference Fig. 9 FIG. 1 is a flow chart of a method for panorama segmentation of a video provided by an embodiment of the present application. Optionally, the method can be applied to the aforementioned target segmentation system, for example Figure 1 The target segmentation system shown in Fig. 9 The video panoramic segmentation method shown may include steps 901-904. It should be understood that for the convenience of description, this application is described in the order of 901-904, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. The following description takes the execution subject of steps 901-904 of the video panoramic segmentation method as a server as an example, and this application is also applicable to other execution subjects. Steps 901-904 are as follows:

[0159] 901. Acquire a video frame, and perform initial segmentation processing on the video frame to obtain a first foreground image and a first background image in the video frame.

[0160] It can be understood that the first background image is an image of uncountable objects in panoramic segmentation, such as lanes, etc. The first foreground image is an image of countable objects in panoramic segmentation, such as pedestrians, vehicles, etc.

[0161] The initial segmentation may be, for example, a video panoramic segmentation based on a SAM model or the like.

[0162] 902. Process the video frame, the first foreground image, and the multi-scale prompt features of the first foreground image through a first target segmentation network to obtain a second foreground image.

[0163] The second foreground image has a higher segmentation accuracy than the first foreground image. The second foreground image corresponds to the first foreground image.

[0164] For example, the second foreground image may be a bicycle A with a relatively high segmentation accuracy, and the first foreground image may be a bicycle A with a relatively low segmentation accuracy.

[0165] The first target segmentation network (such as Fig.10 The general segmentation framework 1 and the general segmentation framework 2 shown in the figure can be, for example, Figure 3 The target segmentation network shown. In one possible implementation, step 902 may include:

[0166] Encoding the video frame to obtain an encoded video frame;

[0167] Performing semantic mask decoding processing on the encoded video frame to obtain a semantic mask of the encoded video frame, a category prediction result of a target foreground image in the encoded video frame, and a semantic feature of the encoded video frame;

[0168] A mask of the target foreground image is obtained by performing processing based on the multi-scale prompt features of the target foreground image, the encoded image, and the semantic features of the encoded image.

[0169] For an introduction to this part, see Figure 2 The description of the illustrated embodiment will not be repeated here.

[0170] For example, Fig.10 As shown in the things refinement module, the countable object things refinement module in the panoptic segmentation can be used to refine the foreground such as people and cars. Among them, based on the mask, bbox and other multi-scale prompt information corresponding to the first foreground image things, the video frame and the first foreground image things are processed to obtain the second foreground image refinedthings. Among them, as Fig.10 The general segmentation framework 1 and the general segmentation framework 2 shown can be referred to Figure 3 The target segmentation network shown is not described here. In this example, the multi-scale prompt information of the target foreground image can be mask information, bounding box bbox information, etc. Of course, it can also be other information, and this solution does not limit this.

[0171] 903. Process the first background image and the prompt features of the first background image through a second object segmentation network to obtain a reference mask of the first background image.

[0172] Reference Fig.10 As shown in the stuff compensation module, the uncountable object stuff compensation module in the panoramic segmentation is used to optimize the background such as the lane. The mask prompt feature of the first background image stuff and the first background image stuff are input into the general segmentation framework 3 for processing, and the reference mask of the first background image, that is, the prediction result of stuff, can be obtained. The specific structure of the general segmentation framework 3 can be found in Figure 3 For this processing process, please refer to the aforementioned Figure 2 The description of the illustrated embodiment will not be repeated here.

[0173] 904. Perform feature fusion processing based on the reference mask of the first background image, the first background image, and the second foreground image to obtain a second background image, wherein the segmentation accuracy corresponding to the second background image is greater than that of the first background image.

[0174] The second background image corresponds to the first background image.

[0175] For example, refer to Fig.10 As shown in the stuff compensation module, based on the first background image stuff and its reference mask and the second foreground image refined things, the stuff feature fusion process is performed to obtain the second background image refined stuff. The segmentation accuracy of the second background image is greater than that of the first background image.

[0176] Based on the above processing, a relatively high-precision panoramic segmentation target can be obtained.

[0177] In a possible implementation manner, a temporal feature matching process is performed based on the second foreground image, the second background image, the third foreground image, and the third background image to obtain a fourth foreground image and a fourth background image.

[0178] Among them, the third foreground image and the third background image are both obtained by processing the video frame through a third object segmentation network.

[0179] The segmentation accuracy of the fourth foreground image is greater than that of the second foreground image and the third foreground image. The segmentation accuracy of the fourth background image is greater than that of the second background image and the third background image.

[0180] Exemplarily, the fourth foreground image and the fourth background image are both video-level images.

[0181] That is to say, based on steps 901-904, the segmentation accuracy can be further improved by performing matching processing based on the time series features.

[0182] For example, Fig.10 As shown in the temporal feature matching module, the temporal feature matching module is used to introduce universal features to facilitate temporal feature matching. Among them, by processing the video frame based on the universal segmentation framework 4 (i.e., the third target segmentation network), a third foreground image and a third background image can be obtained. Among them, the specific structure of the universal segmentation framework 3 can be found in Figure 3Based on the second background image refinedstuff and the third background image, pixel-level matching processing can be performed to obtain the video-level stuff (i.e., the fourth background image). Correspondingly, based on the second foreground image refinedthings and the third foreground image, pixel-level matching processing can be performed to obtain the video-level things (i.e., the fourth foreground image).

[0183] It can be understood that the first foreground image, the second foreground image, the third foreground image and the fourth foreground image correspond to each other. That is, they may correspond to different segmentation accuracies of the same object things. The first background image, the second background image, the third background image and the fourth background image correspond to each other. That is, they may correspond to different segmentation accuracies of the same object stuff.

[0184] This embodiment optimizes things and stuff using multiple general segmentation frameworks based on the existing panoramic segmentation framework, and optimizes multi-target tracking in combination with a temporal feature matching module. The modules work together to improve the effect of panoramic segmentation.

[0185] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0186] The above detailed description of the method of the embodiment of the present application, the following provides a device of the embodiment of the present application. It can be understood that in the various device embodiments of the present application, the division of multiple units or modules is only a logical division according to function, and is not used as a limitation on the specific structure of the device. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may also be combined into one functional module, but no matter whether these functional modules are subdivided or combined, the general process performed by the device is the same. For example, some devices contain a receiving unit and a sending unit. In some designs, the sending unit and the receiving unit can also be integrated into a communication unit, which can implement the functions implemented by the receiving unit and the sending unit. Usually, each unit corresponds to its own program code (or program instruction), and when the program code corresponding to each of these units is run on the processor, the unit is controlled by the processing unit to execute the corresponding process to implement the corresponding function.

[0187] The present application also provides a device for implementing any of the above methods. For example, a target segmentation device is provided, including: Figure 2The modules (or means) of each step executed by the server in any of the methods in the illustrated embodiments. For example, another video panorama segmentation device is provided, including a method for implementing the above Fig. 9 The modules (or means) of each step executed by the server in any method of the illustrated embodiments.

[0188] For example, refer to Fig.11 FIG. 1 is a schematic diagram of the structure of a target segmentation device provided in an embodiment of the present application. The target segmentation device is used to implement the aforementioned target segmentation method, for example Figure 2 The target segmentation method shown.

[0189] like Fig.11 As shown, the device may include an encoding module 1101, a processing module 1102 and a segmentation module 1103, which are specifically as follows:

[0190] The encoding module 1101 is used to acquire an image and perform encoding processing on the image to obtain an encoded image;

[0191] The processing module 1102 is used to perform semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image;

[0192] The segmentation module 1103 is used to obtain a mask of the target object based on the prompt features of the target object in the image, the encoded image, and the semantic features of the encoded image. The prompt features of the target object are obtained based on the prompt information of the target object.

[0193] In a possible implementation, the segmentation module 1103 is used to:

[0194] Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object;

[0195] Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object;

[0196] Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

[0197] In a possible implementation, a prompt module is also included, which is further used to:

[0198] Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object;

[0199] Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images;

[0200] Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively;

[0201] The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

[0202] In a possible implementation, the prompt module is further used to:

[0203] Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features;

[0204] Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

[0205] In a possible implementation, the processing module 1102 is a semantic mask decoding module, wherein the semantic mask decoding module includes a first transformer structure, an upsampling layer, and a feedforward neural network.

[0206] In one possible implementation, the segmentation module 1103 is a category-free mask decoding module, wherein the category-free mask decoding module includes a second transformer structure, a third transformer structure, a token-image feature transition attention layer, an upsampling layer and a feedforward neural network.

[0207] In a possible implementation, the prompt module is a multi-scale prompt encoding module, and the multi-scale prompt encoding module includes a progressive refinement prompt unit, a multi-scale prompt encoding unit and a multi-scale prompt adapter.

[0208] For example, refer to Fig.12 FIG. 1 is a schematic diagram of the structure of a video panorama segmentation device provided in an embodiment of the present application. The video panorama segmentation device is used to implement the aforementioned video panorama segmentation method, for example Fig. 9 The video panoptic segmentation method shown.

[0209] like Fig.12As shown, the device may include a first processing module 1201, a second processing module 1202, a third processing module 1203 and a fourth processing module 1204, which are specifically as follows:

[0210] The first processing module 1201 is used to obtain a video frame and perform initial segmentation processing on the video frame to obtain a first foreground image and a first background image in the video frame;

[0211] A second processing module 1202 is used to process the video frame, the first foreground image and the multi-scale prompt feature of the first foreground image through a first target segmentation network to obtain a second foreground image, the segmentation accuracy corresponding to the second foreground image is greater than that of the first foreground image, and the second foreground image corresponds to the first foreground image;

[0212] A third processing module 1203 is used to process the first background image and the prompt feature of the first background image through a second object segmentation network to obtain a reference mask of the first background image;

[0213] A fourth processing module 1204 is configured to perform feature fusion processing based on the reference mask of the first background image, the first background image, and the second foreground image to obtain a second background image, wherein the segmentation accuracy corresponding to the second background image is greater than that of the first background image, and the second background image corresponds to the first background image;

[0214] Wherein, the first target segmentation network and / or the second target segmentation network are as follows Figure 3 The network shown.

[0215] In a possible implementation manner, the apparatus further includes a fifth processing module, configured to:

[0216] Based on the second foreground image, the second background image, the third foreground image, and the third background image, temporal feature matching processing is performed to obtain a fourth foreground image and a fourth background image, wherein the third foreground image and the third background image are both obtained by processing the video frame through a third target segmentation network, the second foreground image, the third foreground image, and the fourth foreground image correspond to each other, the second background image, the third background image, and the fourth background image correspond to each other, the segmentation accuracy of the fourth foreground image is greater than that of the second foreground image and the third foreground image, and the segmentation accuracy of the fourth background image is greater than that of the second background image and the third background image.

[0217] It should be understood that the division of each module in each of the above devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in each of the above devices can be implemented in the form of a processor calling software; for example, the target segmentation device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each module of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, so as to realize the functions of some or all of the above units. All modules of the above devices can be realized in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the rest by hardware circuits.

[0218] Reference Fig.13 As shown in FIG. 1 , it is a schematic diagram of the hardware structure of another target segmentation device provided in an embodiment of the present application. Fig.13 The target segmentation device 1300 shown (the device 1300 may be a computer device) includes a memory 1301, a processor 1302, a communication interface 1303 and a bus 1304. The memory 1301, the processor 1302 and the communication interface 1303 are connected to each other through the bus 1304.

[0219] The memory 1301 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0220] The memory 1301 can store programs. When the program stored in the memory 1301 is executed by the processor 1302, the processor 1302 and the communication interface 1303 are used to execute the various steps of the target segmentation method of the embodiment of the present application.

[0221] The processor 1302 is a circuit with signal processing capability. In one implementation, the processor 1302 may be a circuit with instruction reading and running capability, such as a central processing unit CPU, a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP), etc.; in another implementation, the processor 1302 may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as the processor 1302 being a hardware circuit implemented by an ASIC or a programmable logic device PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and implementing the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The processor 1302 is used to execute relevant programs to implement the functions that need to be performed by the units in the target segmentation device of the embodiment of the present application, or to execute the target segmentation method of the method embodiment of the present application.

[0222] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0223] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0224] The communication interface 1303 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1300 and other devices or a communication network. For example, data can be acquired through the communication interface 1303 .

[0225] The bus 1304 may include a path for transmitting information between various components of the device 1300 (eg, the memory 1301 , the processor 1302 , and the communication interface 1303 ).

[0226] It should be noted that although Fig.13 The device 1300 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 1300 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 1300 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 1300 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig.13 All devices shown in .

[0227] Reference Fig.14 , which is a hardware structure diagram of another video panorama segmentation device provided in an embodiment of the present application. Fig.14 The video panorama segmentation device 1400 (the device 1400 may be a computer device) includes a memory 1401, a processor 1402, a communication interface 1403 and a bus 1404. The memory 1401, the processor 1402 and the communication interface 1403 are connected to each other through the bus 1404.

[0228] The memory 1401 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0229] The memory 1401 can store programs. When the program stored in the memory 1401 is executed by the processor 1402, the processor 1402 and the communication interface 1403 are used to execute the various steps of the video panoramic segmentation method of the embodiment of the present application.

[0230] The processor 1402 is a circuit with signal processing capability. In one implementation, the processor 1402 may be a circuit with instruction reading and running capability, such as a central processing unit CPU, a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP), etc.; in another implementation, the processor 1402 may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as the processor 1402 being a hardware circuit implemented by an ASIC or a programmable logic device PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The processor 1402 is used to execute relevant programs to implement the functions required to be performed by the units in the video panoramic segmentation device of the embodiment of the present application, or to execute the video panoramic segmentation method of the method embodiment of the present application.

[0231] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0232] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0233] The communication interface 1403 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1400 and other devices or a communication network. For example, data can be obtained through the communication interface 1403.

[0234] The bus 1404 may include a path for transmitting information between various components of the device 1400 (eg, the memory 1401 , the processor 1402 , and the communication interface 1403 ).

[0235] It should be noted that although Fig.14 The device 1400 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 1400 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 1400 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 1400 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig.14 All devices shown in .

[0236] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0237] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0238] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0239] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0242] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A target segmentation method, characterized in that: include: Acquire an image, and perform encoding processing on the image to obtain an encoded image; Performing semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image; The mask of the target object is obtained by processing based on the prompt feature of the target object in the image, the encoded image, and the semantic feature of the encoded image, wherein the prompt feature of the target object is obtained based on the prompt information of the target object.

2. The method according to claim 1, characterized in that: The step of processing based on the prompt feature of the target object in the image, the encoded image, and the semantic feature of the encoded image to obtain the mask of the target object includes: Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object; Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object; Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object; Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images; Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively; The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

4. The method according to claim 3, characterized in that The step of processing the encoded scattered point features respectively corresponding to the images of different scales to obtain the prompt features of the target object includes: Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features; Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

5. The method according to claim 3 or 4, characterized in that: The semantic mask of the encoded image, the category prediction result of the target object in the encoded image, and the semantic features of the encoded image are obtained by inputting the encoded image into a semantic mask decoding module for processing, wherein the semantic mask decoding module includes a first transformer structure, an upsampling layer, and a feedforward neural network.

6. The method according to any one of claims 1 to 5, characterized in that: The mask of the target object is obtained by inputting the prompt features of the target object, the encoded image, and the semantic features of the encoded image into a category-free mask decoding module for processing, wherein the category-free mask decoding module includes a second transformer structure, a third transformer structure, a token-image feature transition attention layer, an upsampling layer, and a feedforward neural network.

7. The method according to any one of claims 3 to 5, characterized in that: The prompt feature of the target object is obtained by inputting the semantic mask, the category prediction result of the target object and the prompt information of the target object into a multi-scale prompt encoding module for processing. The multi-scale prompt encoding module includes a progressive refinement prompt unit, a multi-scale prompt encoding unit and a multi-scale prompt adapter.

8. A target segmentation network, characterized in that The network includes an encoding module, a semantic mask decoding module and a category-free mask decoding module, wherein: The encoding module is used to encode the acquired image to obtain an encoded image; The semantic mask decoding module is used to perform semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image; The category-free mask decoding module is used to obtain a mask of the target object by processing based on the prompt features of the target object in the image, the encoded image, and the semantic features of the encoded image, wherein the prompt features of the target object are obtained based on the prompt information of the target object.

9. The network according to claim 8, characterized in that The classless mask decoding module is used to: Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object; Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object; Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

10. The network according to claim 8 or 9, characterized in that The network also includes a multi-scale cue encoding module for: Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object; Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images; Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively; The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

11. The network according to claim 10, characterized in that The multi-scale prompt encoding module is also used for: Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features; Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

12. A video panorama segmentation method, characterized in that: include: Acquire a video frame, and perform initial segmentation processing on the video frame to obtain a first foreground image and a first background image in the video frame; Processing the video frame, the first foreground image, and the multi-scale prompt features of the first foreground image through a first target segmentation network to obtain a second foreground image, wherein the segmentation accuracy corresponding to the second foreground image is greater than that of the first foreground image, and the second foreground image corresponds to the first foreground image; Processing the first background image and the prompt features of the first background image through a second object segmentation network to obtain a reference mask of the first background image; performing feature fusion processing based on the reference mask of the first background image, the first background image, and the second foreground image to obtain a second background image, wherein the segmentation accuracy corresponding to the second background image is greater than that of the first background image, and the second background image corresponds to the first background image; The first target segmentation network and / or the second target segmentation network is a network as described in any one of claims 8 to 11.

13. The method according to claim 12, characterized in that The method further comprises: Based on the second foreground image, the second background image, the third foreground image, and the third background image, temporal feature matching processing is performed to obtain a fourth foreground image and a fourth background image, wherein the third foreground image and the third background image are both obtained by processing the video frame through a third target segmentation network, the second foreground image, the third foreground image, and the fourth foreground image correspond to each other, the second background image, the third background image, and the fourth background image correspond to each other, the segmentation accuracy of the fourth foreground image is greater than that of the second foreground image and the third foreground image, and the segmentation accuracy of the fourth background image is greater than that of the second background image and the third background image.

14. A target segmentation device, characterized in that: include: The encoding module is used to acquire an image and perform encoding processing on the image to obtain an encoded image; A processing module, used for performing semantic mask decoding processing on the encoded image to obtain semantic features of the encoded image; The segmentation module is used to obtain a mask of the target object by processing based on the prompt features of the target object in the image, the encoded image, and the semantic features of the encoded image, wherein the prompt features of the target object are obtained based on the prompt information of the target object.

15. The device according to claim 14, characterized in that The segmentation module is used for: Processing based on the prompt features of the target object and the semantic features of the encoded image to obtain the contour features of the target object; Processing based on the contour features of the target object, the encoded image, and the semantic features of the encoded image to obtain detail features of the target object; Processing is performed based on the detail features of the target object and the contour features of the target object to obtain a mask of the target object.

16. The device according to claim 14 or 15, characterized in that Also includes prompt module, also used for: Processing images of different scales containing the target object respectively to obtain masks corresponding to the images of different scales, wherein the images of different scales containing the target object include prompt information of the target object; Based on the masks corresponding to the images of different scales, the semantic masks of the encoded images, and the category prediction results of the target object, the scattered sampling features corresponding to the images of different scales are obtained, wherein the semantic masks of the encoded images and the category prediction results of the target object are obtained by processing the encoded images; Performing encoding processing on the scattered sampling features corresponding to the images of different scales, respectively, to obtain encoded scattered features corresponding to the images of different scales, respectively; The encoded scattered point features corresponding to the images of different scales are processed to obtain the prompt features of the target object.

17. The device according to claim 16, characterized in that The prompt module is also used for: Processing the encoded scattered point features corresponding to the small-scale images in the images of different scales to obtain small-scale features; Processing is performed based on the small-scale features and the encoded scattered point features corresponding to the large-scale images in the images of different scales to obtain the prompt features of the target object.

18. The device according to claim 16 or 17, characterized in that The processing module is a semantic mask decoding module, wherein the semantic mask decoding module includes a first transformer structure, an upsampling layer and a feedforward neural network.

19. The device according to any one of claims 14 to 18, characterized in that The segmentation module is a category-free mask decoding module, wherein the category-free mask decoding module includes a second transformer structure, a third transformer structure, a token-image feature transition attention layer, an upsampling layer and a feedforward neural network.

20. The device according to any one of claims 16 to 18, characterized in that The prompt module is a multi-scale prompt encoding module, and the multi-scale prompt encoding module includes a progressive refinement prompt unit, a prompt encoding unit and a multi-scale prompt adapter.

21. A video panorama segmentation device, characterized in that: include: A first processing module, used for acquiring a video frame, and performing initial segmentation processing on the video frame to obtain a first foreground image and a first background image in the video frame; A second processing module is used to process the video frame, the first foreground image and the multi-scale prompt feature of the first foreground image through a first target segmentation network to obtain a second foreground image, the segmentation accuracy corresponding to the second foreground image is greater than that of the first foreground image, and the second foreground image corresponds to the first foreground image; a third processing module, configured to process the first background image and the prompt feature of the first background image through a second object segmentation network to obtain a reference mask of the first background image; a fourth processing module, configured to perform feature fusion processing based on a reference mask of the first background image, the first background image, and the second foreground image to obtain a second background image, wherein the segmentation accuracy corresponding to the second background image is greater than that of the first background image, and the second background image corresponds to the first background image; The first target segmentation network and / or the second target segmentation network is a network as described in any one of claims 8 to 11.

22. The device according to claim 21, characterized in that The device further comprises a fifth processing module, configured to: Based on the second foreground image, the second background image, the third foreground image, and the third background image, temporal feature matching processing is performed to obtain a fourth foreground image and a fourth background image, wherein the third foreground image and the third background image are both obtained by processing the video frame through a third target segmentation network, the second foreground image, the third foreground image, and the fourth foreground image correspond to each other, the second background image, the third background image, and the fourth background image correspond to each other, the segmentation accuracy of the fourth foreground image is greater than that of the second foreground image and the third foreground image, and the segmentation accuracy of the fourth background image is greater than that of the second background image and the third background image.

23. A target segmentation device, characterized in that: It comprises a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1 to 7.

24. A video panorama segmentation device, characterized in that: It comprises a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method as claimed in claim 12 or 13.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 7, or the method according to claim 12 or 13.

26. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 7, or the method according to claim 12 or 13.

27. A vehicle, characterized in that: The vehicle comprises the object segmentation device according to any one of claims 14 to 20 or the video panoramic segmentation device according to claim 21 or 22.

Images