Image instance segmentation method and device, equipment, storage medium and program product

Abstract

The invention discloses an image instance segmentation method and device, equipment, a storage medium and a program product, and relates to the field of image processing. The method comprises the following steps: acquiring a sample image, wherein the sample image comprises a to-be-identified sample instance; predicting instance contours of the sample instances through the image segmentation model to obtain predicted instance contours; taking a predicted reference position point in the predicted instance contour as a pole to obtain predicted contour polar coordinates corresponding to the plurality of contour points respectively; and obtaining a loss value based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, and training to obtain a trained image segmentation model. Through the above mode, the instance contour can be expressed more meticulously and visually by introducing the polar coordinates, and then targeted training is performed on the image segmentation model, so that the accuracy of instance contour identification is improved, and the accuracy and segmentation efficiency of instance segmentation are improved. The method can be applied to various scenes such as cloud technology, artificial intelligence and intelligent transportation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a method, apparatus, device, storage medium and program product for segmenting an image instance. Background Art

[0002] With the development of computer technology, performance testing of various applications and terminals has become increasingly important. Performance testing is used to identify the correctness, security and quality of program operation. Good performance testing is conducive to more accurately measuring the program operation status, so as to more promptly find defects (bugs) missed or generated during the program development process, so as to better repair and improve the program.

[0003] In the related art, when testing a game program installed in a terminal, the server usually receives game images uploaded by multiple terminals. When the tester manually clicks on the main test machine among the multiple terminals, based on the control position clicked by the tester, a fixed-size image area centered on the control position is extracted from the test image corresponding to the main test machine. Using the image area as a template, multi-scale template matching is performed on the game images uploaded by other terminals, so as to determine the click position with a high overlap ratio with the template from different terminals and perform an automatic triggering process, so as to achieve the purpose of jointly testing multiple terminals based on the main test machine.

[0004] In the above process, although the test efficiency of joint testing of multiple terminals can be improved by operating the main test machine, the analysis of controls in this method is relatively generalized. When the resolution of two compared images differs greatly, it is difficult to achieve a good matching effect by comparing the overlap ratio, which will lead to poor test accuracy in performance testing of multiple terminals. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment, storage medium and program product for segmenting an image instance, which can introduce polar coordinates to express the instance contour in a more detailed and intuitive manner, and then use loss values ​​to conduct targeted training on the image segmentation model, thereby obtaining an image segmentation model that is more accurate in identifying the instance contour, thereby improving the accuracy and efficiency of instance segmentation. The technical solution is as follows.

[0006] In one aspect, a method for segmenting an image instance is provided, the method comprising:

[0007] Acquire a sample image, wherein the sample image includes a sample instance to be identified, and the sample instance is annotated with sample contour polar coordinates for characterizing the instance contour;

[0008] Predicting the instance contour of the sample instance in the sample image by using an image segmentation model to obtain a predicted instance contour corresponding to the sample instance, wherein the image segmentation model is a model to be trained;

[0009] Taking the predicted reference position point in the predicted instance contour as the pole, obtaining the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour;

[0010] Based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtaining a loss value corresponding to the sample instance;

[0011] The image segmentation model is trained with the loss value to obtain a trained image segmentation model.

[0012] In another aspect, a device for segmenting an image instance is provided, the device comprising:

[0013] An image acquisition module, used for acquiring a sample image, wherein the sample image includes a sample instance to be identified, and the sample instance is annotated with sample contour polar coordinates for characterizing the instance contour;

[0014] A contour prediction module, used for predicting the instance contour of the sample instance in the sample image by using an image segmentation model to obtain a predicted instance contour corresponding to the sample instance, wherein the image segmentation model is a model to be trained;

[0015] A coordinate prediction module, used to obtain the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour, taking the predicted reference position point in the predicted instance contour as a pole;

[0016] A loss acquisition module, configured to acquire a loss value corresponding to the sample instance based on a difference between the sample contour polar coordinates and the predicted contour polar coordinates;

[0017] The model training module is used to train the image segmentation model with the loss value to obtain a trained image segmentation model.

[0018] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method for segmenting an image instance as described in any of the above-mentioned embodiments of the present application.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a method for segmenting an image instance as described in any of the above-mentioned embodiments of the present application.

[0020] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image instance segmentation method described in any of the above embodiments.

[0021] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0022] A sample image including a sample instance is obtained, and the sample instance in the sample image is annotated with the sample contour polar coordinates for characterizing the instance contour; the predicted instance contour is obtained through the image segmentation model, and the predicted contour polar coordinates are determined with the predicted reference position point in the predicted instance contour as the pole, so as to obtain the loss value corresponding to the sample instance based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates. On the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more meticulously and intuitively by introducing polar coordinates, and the sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance are combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in instance contour recognition, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0025] Figure 2 is a flow chart of a method for segmenting an image instance provided by an exemplary embodiment of the present application;

[0026] Figure 3 is a flow chart of determining sample contour polar coordinates and predicted contour polar coordinates provided by an exemplary embodiment of the present application;

[0027] Figure 4 is a schematic diagram of determining an instance bounding box and polar coordinates of a sample contour provided by an exemplary embodiment of the present application;

[0028] Figure 5 is a flow chart of determining a loss value based on a sample contour polar coordinate and a predicted contour polar coordinate provided by an exemplary embodiment of the present application;

[0029] Figure 6 is a flow chart of determining a loss value through a contour deviation value and a coordinate deviation value provided by an exemplary embodiment of the present application;

[0030] Figure 7 is a flow chart of determining a loss value through a contour deviation value, a coordinate deviation value and a size deviation value provided by an exemplary embodiment of the present application;

[0031] Figure 8 is a flow chart of determining a loss value through sample feature maps of different feature map sizes provided by an exemplary embodiment of the present application;

[0032] Fig. 9 is a schematic diagram of a model structure of an image segmentation model provided by an exemplary embodiment of the present application;

[0033] Fig.10 is a schematic diagram of determining a first sample feature grid based on a first sample feature map provided by an exemplary embodiment of the present application;

[0034] Fig.11 is a schematic diagram of determining a first sample feature grid based on a second sample feature map provided by an exemplary embodiment of the present application;

[0035] Fig.12 is a flow chart of applying an image segmentation model provided by an exemplary embodiment of the present application;

[0036] Fig.13 is a flow chart of a method for segmenting an image instance provided by another exemplary embodiment of the present application;

[0037] Fig.14 is a schematic diagram of a game screenshot provided by an exemplary embodiment of the present application;

[0038] Fig.15 is a schematic diagram of a game screenshot provided by another exemplary embodiment of the present application;

[0039] Fig.16is a schematic diagram of different pixel-level annotations of game controls in a game screenshot provided by an exemplary embodiment of the present application;

[0040] Fig.17 is a structural block diagram of an image instance segmentation device provided by an exemplary embodiment of the present application;

[0041] Fig.18 is a structural block diagram of an image instance segmentation device provided by another exemplary embodiment of the present application;

[0042] Fig.19 It is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0045] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0046] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0047] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0048] In the related art, when testing a game program installed in a terminal, the server usually receives game images uploaded by multiple terminals. When the tester manually clicks on the main test machine among the multiple terminals, based on the control position clicked by the tester, an image area of ​​fixed size centered on the control position is extracted from the test image corresponding to the main test machine. Using the image area as a template, multi-scale template matching is performed on the game images uploaded by other terminals, so as to determine the click position with high similarity to the template from different terminals and perform an automatic triggering process, so as to achieve the purpose of jointly testing multiple terminals based on the main test machine. In the above process, although the test efficiency of jointly testing multiple terminals can be improved by operating the main test machine, the method is relatively generalized in the analysis of controls. When the size, shape and other attributes of the controls are similar, it is difficult to achieve a good matching effect, which will lead to poor test accuracy of performance testing of multiple terminals.

[0049] In an embodiment of the present application, a method for segmenting an image instance is provided, which can express the instance contour in a more detailed and intuitive manner by introducing polar coordinates, and then use loss values ​​to perform targeted training on the image segmentation model, thereby obtaining an image segmentation model that is more accurate in identifying the instance contour, thereby improving the accuracy of instance segmentation and the efficiency of segmentation. The method for segmenting an image instance provided by the present application includes at least one of a variety of image recognition scenarios, such as image instance segmentation scenarios, image content recognition scenarios, and multi-machine testing scenarios, when applied.

[0050] It is worth noting that the above application scenarios are only illustrative examples, and the image instance segmentation method provided in this embodiment can also be applied to other scenarios, which is not limited in this embodiment of the present application.

[0051] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions. For example, the sample instances, image segmentation models and other contents involved in this application are all obtained with full authorization.

[0052] Secondly, the implementation environment involved in the embodiments of the present application is described. The image instance segmentation method provided in the embodiments of the present application can be implemented by a terminal alone, or by a server, or by a terminal and a server through data interaction, which is not limited in the embodiments of the present application. Optionally, the segmentation method of an image instance executed interactively by a terminal and a server is used as an example for description.

[0053] For illustration, please refer to Figure 1 The implementation environment involves a terminal 110 and a server 120 , and the terminal 110 and the server 120 are connected via a communication network 130 .

[0054] In some embodiments, the terminal 110 has an image acquisition function, which is used to acquire an image by at least one of the image acquisition methods such as shooting, screenshot, and downloading. Optionally, the terminal 110 can also receive an instance annotation operation to annotate the shape of the instance in the image. For example, annotate the shape of the key content (such as animals, buildings, etc.) in the image; or annotate the shape of the functional control (such as a control for performing a virtual attack) in the image.

[0055] Illustratively, after the terminal 110 obtains an image through at least one of the above-mentioned image acquisition methods, it receives an instance labeling operation for an instance therein, and the instance labeling operation is implemented by labeling the sample instance with sample contour polar coordinates, and using the labeled image as the sample image, and the instance to be labeled as the sample instance 141 to be identified. The sample instance 141 is labeled with sample contour polar coordinates for characterizing the instance contour, thereby describing the shape (instance contour) of the sample instance 141 in the form of data.

[0056] Optionally, the terminal 110 sends the sample image to the server 120 through the communication network 130. The server 120 is configured with an image segmentation model, which is a model to be trained and is used to train to obtain a trained image segmentation model 121. After the server 120 obtains the sample image, the server 120 predicts the instance contour of the sample instance 141 in the sample image through the image segmentation model, and obtains the predicted instance contour, and then takes the predicted reference position point in the predicted instance contour as the pole, and obtains the predicted contour polar coordinates corresponding to multiple contour points on the predicted instance contour. Among them, the predicted reference position point is a point in the predicted instance contour. When the predicted reference position point is taken as the pole, the corresponding predicted contour polar coordinates can be determined based on multiple predicted instance contour points, and the predicted contour polar coordinates are used to present the predicted instance contour in data form.

[0057] In some embodiments, the server 120 obtains the loss value corresponding to the sample instance 141 based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, thereby training the image segmentation model with the loss value and obtaining the image segmentation model 121. The image segmentation model 121 is used to perform segmentation prediction on the image instance in the image to be identified and obtain the instance prediction result.

[0058] Optionally, the terminal 110 sends a large number of sample images to the server 120. After the server 120 obtains the image segmentation model 121 through training, the terminal 110 sends the image to be identified to the server 120, so that the server 120 predicts the image instance in the image to be identified with the help of the image segmentation model 121 to obtain the instance prediction result. Schematically, the server 120 sends the instance prediction result to the terminal 110 so that the terminal 110 renders and displays the image corresponding to the instance prediction result based on the instance prediction result, or so that the terminal 110 triggers a corresponding instruction based on the instance prediction result.

[0059] In an optional embodiment, when the server 120 or the terminal 110 independently executes the above-mentioned image instance segmentation method, the server 120 or the terminal 110 obtains the sample image by itself, and obtains the trained image segmentation model 121 for predicting the image to be recognized through the training process of the image segmentation model configured by itself. The above process is an example of a non-unique situation of the training process of the image segmentation model 121.

[0060] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, and can also be implemented as desktop computers, etc.; the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0061] Among them, cloud technology refers to a hosting technology that unifies hardware, application programs, networks and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient.

[0062] In some embodiments, the above server can also be implemented as a node in a blockchain system.

[0063] Combined with the above-mentioned noun introduction and application scenarios, the image instance segmentation method provided by this application is described, and the method is applied to a server as an example. Figure 2 As shown, the method includes the following steps 210 to 250.

[0064] Step 210: Acquire a sample image.

[0065] Optionally, the sample image is an image obtained by at least one image acquisition method such as photographing, screenshotting, downloading, etc., and the sample image includes a sample instance to be identified.

[0066] In an illustrative manner, the sample instance is the key content in the sample image. For example, the sample image is a photo of a park, and the sample instance is the "animal" that needs to be recognized; or, the sample instance is the selected content in the sample image. For example, the sample image is a screenshot of a game, and the sample instance is the function control therein.

[0067] The sample instance is annotated with sample contour polar coordinates for representing the instance contour, that is, the instance contour of the sample instance can be represented by the sample contour polar coordinates in data form.

[0068] Schematically, the sample instance has a certain shape in the sample image. In order to describe the instance contour of the sample instance that represents the shape, the sample instance is annotated by the sample contour polar coordinates. For example, the instance contour is implemented as an edge curve that depicts the edge of the sample instance, and the sample contour polar coordinates are implemented as polar coordinates corresponding to multiple edge points on the edge curve; or, the instance contour is implemented as a set of pixel points that depict the edge of the sample instance, and the sample contour polar coordinates are implemented as polar coordinates corresponding to each pixel point in the pixel point set, etc.

[0069] Step 220 , predicting the instance contour of the sample instance in the sample image by using the image segmentation model to obtain the predicted instance contour corresponding to the sample instance.

[0070] The image segmentation model is a model to be trained, and the image segmentation model has a certain image segmentation function.

[0071] Illustratively, after obtaining the sample image, the sample image is input into the image segmentation model, so that the instance contour of the sample instance in the sample image is predicted by the image segmentation model.

[0072] Optionally, in the process of prediction using the image segmentation model, a predicted instance contour of the sample instance in the sample image is first preliminarily predicted, and the predicted instance contour is used to predict a rough shape of the sample instance in the sample image.

[0073] Step 230 , taking the predicted reference position point in the predicted instance contour as the pole, obtaining the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour.

[0074] In some embodiments, the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour are acquired through preset polar coordinate acquisition conditions.

[0075] The preset polar coordinate acquisition condition is a preset polar coordinate acquisition condition used to acquire a plurality of predicted contour polar coordinates used to describe the predicted instance contour from the predicted instance contour.

[0076] Optionally, the preset polar coordinate acquisition condition is implemented as an acquisition condition when acquiring the sample contour polar coordinates for the instance contour; or, the preset polar coordinate acquisition condition is implemented as determining multiple contour points from the predicted instance contour at a preset interval angle, and the predicted contour polar coordinates corresponding to the multiple contour points respectively; or, the preset polar coordinate acquisition condition is implemented as randomly determining at least two contour points from the predicted instance contour, and the predicted contour polar coordinates corresponding to the at least two contour points respectively, etc.

[0077] In an optional embodiment, the preset polar coordinate acquisition condition is implemented as a process of determining the polar coordinates of the predicted contour by emitting multiple rays.

[0078] In some embodiments, a plurality of rays are emitted in different directions of the plane where the sample image is located, and the polar coordinates of the predicted contour are determined based on the intersection points of the plurality of rays with the instance contour.

[0079] Schematically, the prediction reference position point is a position point in the prediction instance contour. Optionally, the prediction reference position point is a position point randomly selected from the prediction instance contour; or, the prediction reference position point is a position point predetermined within the prediction instance contour; or, the prediction reference position point is a position point determined based on the distribution of the prediction instance contour in the sample image, etc.

[0080] After determining the predicted reference position point, multiple rays are emitted in different directions of the plane where the sample image is located with the reference position as the pole.

[0081] Optionally, a ray is randomly emitted in multiple directions; or, multiple rays are randomly emitted in multiple directions, wherein the number of rays emitted in different directions may be the same or different; or, a ray is emitted in multiple directions at a preset interval angle (such as: a ray is emitted in 36 directions at a preset interval angle of 10°, etc.); or, multiple rays are emitted in multiple directions at a preset interval angle, etc.

[0082] In some embodiments, based on the emission of multiple rays and the predicted instance contour predicted by the image segmentation model, the intersection points respectively corresponding to the multiple rays and the predicted instance contour are determined, thereby obtaining multiple intersection points.

[0083] Among them, based on predicting the reference position point as a pole, a ray can be selected from multiple rays as a polar axis, so as to determine the intersection polar coordinates corresponding to multiple intersection points based on the pole and the polar axis as the predicted contour polar coordinates.

[0084] Alternatively, based on predicting the reference position point as a pole, a polar axis passing through the pole may be pre-set, so that the intersection polar coordinates corresponding to a plurality of intersection points are determined based on the pole and the polar axis as the predicted contour polar coordinates.

[0085] In an optional embodiment, when the image segmentation model emits multiple rays based on the predicted reference position point, it can emit rays in multiple preset directions based on its own settings. For example, the image segmentation model can emit rays in multiple directions corresponding to the polar angles corresponding to the polar coordinates of multiple sample contours.

[0086] Step 240 , obtaining a loss value corresponding to the sample instance based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates.

[0087] Illustratively, after obtaining the sample contour polar coordinates and the predicted contour polar coordinates, the difference between the sample contour polar coordinates and the predicted contour polar coordinates is compared.

[0088] Optionally, based on the polar coordinates represented by polar angle and polar diameter, the sample contour polar angle and the sample contour polar diameter corresponding to the sample contour polar coordinates are determined, and the predicted contour polar angle and the predicted contour polar diameter corresponding to the predicted contour polar coordinates are determined, so as to obtain the loss value corresponding to the sample instance based on the sample contour polar angle, the sample contour polar diameter, the predicted contour polar angle and the predicted contour polar diameter.

[0089] In some embodiments, multiple sample contour polar coordinates and multiple predicted contour polar coordinates are determined based on the multiple emitted rays. When obtaining a loss value corresponding to a sample instance based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, the loss value is determined by combining the multiple sample contour polar coordinates and the multiple predicted contour polar coordinates.

[0090] Step 250: train the image segmentation model using the loss value to obtain a trained image segmentation model.

[0091] In schematic form, after obtaining the loss value corresponding to the sample instance, the image segmentation model is trained using the loss value, and with the goal of minimizing the loss value, the parameters of the image segmentation model are adjusted using the gradient descent method. When the loss value reaches the minimum (e.g., the current loss value is the same as the most recently calculated loss value or the difference is less than a preset threshold), the training process of the image segmentation model is terminated to obtain the image segmentation model.

[0092] In some embodiments, the sample image includes multiple sample instances, and the loss value corresponding to each sample instance is determined respectively to obtain multiple loss values. The image segmentation model is trained with the help of the multiple loss values ​​to obtain a trained image segmentation model.

[0093] In some embodiments, multiple sample images are obtained, each sample image includes at least one sample instance, and the loss value corresponding to each sample instance is determined respectively, thereby obtaining multiple loss values, and then the image segmentation model is trained with the help of the multiple loss values ​​to obtain a trained image segmentation model.

[0094] Among them, the image segmentation model is used to perform segmentation prediction on image instances in the image to be identified and obtain instance prediction results.

[0095] Illustratively, after the image segmentation model is obtained, an image to be identified is acquired and used as an input of the image segmentation model, and an instance prediction result is obtained by predicting the image instance.

[0096] Optionally, when the image to be identified does not include an image instance, the instance prediction result is implemented as first prompt information that does not include the image instance; or, when the image to be identified includes an image instance, the instance prediction result is implemented as second prompt information that includes the image instance; or, when the image to be identified includes an image instance, the instance prediction result is implemented as annotation information for annotating the image instance, etc.

[0097] For example: when the image to be recognized includes multiple image instances, after the image segmentation model predicts the image instance, it outputs annotated images that respectively annotate the multiple image instances with different annotation information as the instance prediction result; or, when the image to be recognized includes multiple image instances, after the image segmentation model predicts the image instance, it outputs image segmentation results that segment the multiple image instances from the image to be recognized as the instance prediction result, etc.

[0098] The image segmentation model is a model obtained by analyzing and learning the instance contours of the sample instance. Therefore, the image segmentation model can predict and analyze the shape of the sample instance more accurately.

[0099] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0100] In summary, a sample image including a sample instance is obtained, and the sample instance in the sample image is annotated with a sample contour polar coordinate for characterizing the instance contour; the predicted instance contour is obtained through an image segmentation model, and the predicted contour polar coordinate is determined with the predicted reference position point in the predicted instance contour as the pole, so as to obtain the loss value corresponding to the sample instance based on the difference between the sample contour polar coordinate and the predicted contour polar coordinate. On the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more meticulously and intuitively by introducing polar coordinates, and the sample contour polar coordinate obtained by analyzing the sample instance and the predicted contour polar coordinate obtained by predicting the sample instance are combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in instance contour recognition, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation.

[0101] In an optional embodiment, the sample contour polar coordinates annotated by the sample instance are polar coordinates determined by emitting multiple rays in different directions of the plane where the sample image is located with the reference position point in the sample instance as the pole, and when emitting multiple rays with the predicted reference position point in the predicted instance contour as the pole, the predicted contour polar coordinates are determined based on the multiple rays emitted when determining the sample contour polar coordinates. Schematically, as Figure 3 As shown above Figure 2The illustrated steps 210 to 230 may also be implemented as the following steps 310 to 370 .

[0102] Step 310: Acquire a sample image.

[0103] The sample image includes a sample instance to be identified.

[0104] In some embodiments, the instance contour of the sample instance in the sample image is determined by the entirety of the sample instance.

[0105] Optionally, a pixel point corresponding to the sample instance is determined as an instance pixel point; and a value assignment operation is performed on the instance pixel point to determine an instance contour corresponding to the sample instance.

[0106] Schematically, the sample instance in the sample image is predetermined content. Based on the sample instance, multiple pixel points corresponding to the sample instance can be determined from multiple image pixel points corresponding to the sample image. The multiple pixel points are regarded as an instance pixel point set. Through the instance pixel point set, the coordinate position, size and shape corresponding to the sample instance relative to the sample image and other attribute information can be roughly determined.

[0107] Optionally, a pixel value is selected to perform an assignment operation on multiple pixel points in the instance pixel point set, and the selected pixel value can be predetermined or randomly selected. For example: if the sample image is a grayscale image and the predetermined pixel value is 58, then the pixel values ​​of multiple pixel points in the instance pixel point set corresponding to the sample instance are assigned to 58; or, if the sample image is a color image and the predetermined pixel value is (23, 66, 54), then the pixel values ​​of multiple pixel points in the instance pixel point set corresponding to the sample instance are assigned to (23, 66, 54), etc.

[0108] By assigning pixel values, the sample instance is marked to distinguish it from the area outside the sample instance in the sample image, and the instance contour corresponding to the sample instance can be determined based on the boundary between the assigned instance pixel point set and the pixel points outside the sample instance. For example, if there is a difference in the pixel values ​​of the assigned instance pixel point set and the pixel points in the surrounding area, the boundary between the sample instance indicated by the instance pixel point set and the surrounding area is determined based on the difference in pixel value changes, and the boundary is determined as the instance contour of the sample instance.

[0109] In some embodiments, the sample image includes multiple sample instances, and when determining the instance contour, the instance contours corresponding to the multiple sample instances are determined. Optionally, through the assignment operation, different pixel values ​​are used to perform the assignment process for the instance pixel point sets corresponding to the multiple sample instances. For example: the sample image includes sample instance A and sample instance B; the instance pixel point set a corresponding to the sample instance A uses a predetermined first pixel value to perform an assignment operation on multiple pixel values ​​in the instance pixel point set a; the instance pixel point set b corresponding to the sample instance B uses a predetermined second pixel value to perform an assignment operation on multiple pixel values ​​in the instance pixel point set b. The first pixel value and the second pixel value are different, so that different sample instances can be distinguished, and the instance contours corresponding to different sample instances can also be determined.

[0110] In an optional embodiment, the instance contour of the sample instance is determined by performing an instance annotation operation on the sample instance. The instance annotation operation can also be implemented as an annotation of at least one attribute of the sample instance, such as coordinates and size. That is, the instance contour can also be used to characterize information such as the instance position and instance size of the sample instance. This embodiment of the present application is not limited to this.

[0111] Optionally, the image whose instance contour of the sample instance is determined is called a sample image, and the sample image is used to participate in model training in a subsequent process.

[0112] Step 320 , emitting multiple rays in different directions of the plane where the sample image is located with the reference position point in the sample instance as the pole, and determining the polar coordinates of the sample contour based on the intersection points of the multiple rays and the instance contour.

[0113] That is, the polar coordinates of the sample contour are determined by emitting multiple rays in different directions of the plane where the sample image is located with the reference position point in the sample instance as the pole.

[0114] Illustratively, the reference position point is a position point in the sample instance. Optionally, the reference position point is a position point randomly selected from the sample instance; or, the reference position point is a position point predetermined in the sample instance; or, the reference position point is a position point determined based on the instance contour of the sample instance, etc.

[0115] After the reference position point is determined, multiple rays are emitted in different directions of the plane where the sample image is located with the reference position as the pole.

[0116] Optionally, a ray is randomly emitted in multiple directions; or, multiple rays are randomly emitted in multiple directions, wherein the number of rays emitted in different directions may be the same or different; or, a ray is emitted in multiple directions at a preset interval angle (such as: a ray is emitted in 36 directions at a preset interval angle of 10°, etc.); or, multiple rays are emitted in multiple directions at a preset interval angle, etc.

[0117] In some embodiments, based on the emission of multiple rays and the instance contour corresponding to the pre-labeled sample instance, the intersection points respectively corresponding to the multiple rays and the instance contour are determined, thereby obtaining multiple intersection points.

[0118] Among them, based on the reference position point as the pole, a ray can be selected from multiple rays as the polar axis, so as to determine the intersection polar coordinates corresponding to multiple intersection points based on the pole and the polar axis as the sample contour polar coordinates.

[0119] Alternatively, based on the reference position point being the pole, a polar axis passing through the pole may be preset, so that the intersection polar coordinates corresponding to a plurality of intersection points are determined based on the pole and the polar axis as the sample contour polar coordinates.

[0120] Optionally, the sample contour polar coordinates are polar coordinates determined by pre-labeled instance contours and have a certain label reference value.

[0121] In an optional embodiment, an instance bounding box surrounding the sample instance is determined from the sample image.

[0122] Illustratively, after determining the sample instance in the sample image, an image region surrounding the sample instance is determined from the sample image as an instance bounding box.

[0123] Optionally, the example bounding box is implemented in a rectangular shape; or, the example bounding box is implemented in a circular shape; or, the example bounding box is implemented in an irregular polygonal shape, etc.

[0124] Illustratively, the instance bounding box is an image region that just surrounds the sample instance; or, the instance bounding box is slightly larger than the image region that just surrounds the sample instance, etc.

[0125] In some embodiments, the example bounding box is implemented as a rectangular shape that just surrounds the sample instance. Figure 4 , which is a schematic diagram of determining an instance bounding box 420 based on a sample instance 410 .

[0126] Optionally, the sample image includes multiple sample instances. For a sample instance 410 among the multiple sample instances, the length and width of an instance bounding box 420 are determined using an instance contour 411 of the sample instance 410. The lower left corner of the instance bounding box 420 is the minimum value of the horizontal and vertical axes of the sample instance 410 (minimum x value and minimum y value), and the upper right corner of the instance bounding box 420 is the maximum value of the horizontal and vertical axes of the sample instance 410 (maximum x value and maximum y value).

[0127] In some embodiments, when the sample image includes multiple sample instances, the instance bounding boxes corresponding to each sample instance are determined respectively in the above manner.

[0128] In an optional embodiment, based on the diagonal of the instance bounding box, the center point of the instance bounding box is determined as the reference position point in the sample instance.

[0129] Illustratively, after obtaining the instance bounding box corresponding to the sample instance, the bounding box diagonal corresponding to the instance bounding box is determined, and then the center point of the instance bounding box is determined based on at least two bounding box diagonals.

[0130] For example: when the instance bounding box is implemented as a rectangular shape, the two diagonal lines corresponding to the rectangular shape are determined as the diagonals of the bounding box, and the intersection of the two diagonals of the bounding box is determined as the center point of the instance bounding box; or, when the instance bounding box is implemented as a circular shape, at least two diagonal lines (diameters) corresponding to the circular shape are determined as the diagonals of the bounding box, and the intersection of at least two diagonals of the bounding box is determined as the center point of the instance bounding box, etc.

[0131] In some embodiments, the center point of the instance bounding box is used as the reference position point in the sample instance, that is, the reference position point in the sample instance is determined by the instance contour corresponding to the sample instance.

[0132] like Figure 4 As shown, the center point of the instance bounding box 420 is used as the reference position point 421 in the sample instance.

[0133] In an optional embodiment, the reference position point is used as the pole, and rays are emitted in multiple directions of the plane where the sample image is located at preset interval angles.

[0134] The preset interval angle is a preset interval angle, which is used to represent the angle between two adjacent emitted rays. For example, the preset interval angle is 10°, 20°, 15°, etc.

[0135] In some embodiments, when rays are emitted in multiple directions of the plane where the sample image is located at preset interval angles, one ray is emitted in multiple directions of the plane where the sample image is located.

[0136] The multiple rays may be sent simultaneously, sequentially, or at intervals, etc., which is not limited here.

[0137] Illustratively, after obtaining the reference position point, taking the example of emitting one ray in 36 directions of the plane where the sample image is located with a preset interval angle of 10°, the number of emitted rays is 36.

[0138] like Figure 4 As shown, it shows the situation that 8 rays are obtained by emitting one ray respectively in 8 directions of the plane where the sample image is located with the reference position point 421 as the pole, and the angle between two adjacent rays is 45°.

[0139] In an optional embodiment, based on the intersection points between the multiple rays and the instance contour, the polar coordinates of the sample contour corresponding to the multiple contour points are obtained.

[0140] Schematically, there are intersections between the multiple rays and the instance contour, and based on the reference position point as the pole, the polar coordinates of the sample contour corresponding to the sample instance can be determined.

[0141] In an optional embodiment, a polar axis is determined in the sample image based on the pole, and a polar coordinate system is established.

[0142] Indicatively, after determining the pole corresponding to the reference position point, a ray is randomly selected from multiple rays as the polar axis to establish a polar coordinate system; alternatively, a line segment is randomly determined as the polar axis in the sample image based on the pole, and the line segment may or may not coincide with a ray.

[0143] In an optional embodiment, based on the intersection points between the multiple rays and the instance contour, the polar coordinates of the sample contour of the multiple intersection points in the polar coordinate system are obtained.

[0144] Illustratively, after the intersection points between the multiple rays and the instance contour are obtained, multiple intersection points are determined, and based on the angles and positions of the multiple intersection points in the polar coordinate system, the polar coordinates of the sample contour corresponding to the multiple intersection points are determined.

[0145] The polar coordinates of the sample contour are represented by the polar angle of the sample contour and the polar diameter of the sample contour. The polar angle of the sample contour is determined according to the angle between the ray corresponding to the intersection and the polar axis; the polar diameter of the sample contour is determined according to the distance between the intersection and the pole.

[0146] Illustratively, based on the position of the intersection point in the polar coordinate system, the angle between the ray corresponding to the intersection point and the polar axis is determined as the sample contour polar angle in the sample contour polar coordinates.

[0147] Indicatively, Figure 4As shown, after the pole corresponding to the reference position point 421 emits multiple rays, when the ray encounters the instance contour 411 of the sample instance 410, it stops, and determines the intersection point (such as: intersection point 422), calculates the angle between the intersection point 422 and the polar axis (such as: taking the ray 412 as the polar axis), and the angle is determined in the counterclockwise direction with the polar axis as the starting axis, so as to obtain the sample contour polar angle in the sample contour polar coordinates corresponding to the intersection point 422. The angle can also be an angle determined in the clockwise direction, which is not limited here.

[0148] Similarly, the polar angles of the sample contour corresponding to the intersection points of the multiple rays and the instance contour are obtained through the above angle determination method.

[0149] The intersection distance (length) from the reference position point 421 to the intersection point 422 has a maximum value of half the length of the diagonal of the instance bounding box. The intersection distance is divided by half the length of the diagonal of the instance bounding box, thereby realizing the process of normalizing the intersection distance based on the length of the diagonal of the bounding box, and obtaining the sample contour polar diameter in the sample contour polar coordinates corresponding to the intersection point 422.

[0150] In some embodiments, the distance between the intersection and the pole is determined to obtain the intersection distance; the diagonal length of the bounding box corresponding to the instance bounding box is obtained; the intersection distance is normalized based on the diagonal length of the bounding box to obtain the sample contour polar diameter in the sample contour polar coordinates.

[0151] Indicatively, Figure 4 As shown, after the pole corresponding to the reference position point 421 emits multiple rays, the rays stop when they encounter the instance contour 411 of the sample instance 410, and determine the intersection point (such as intersection 422), and calculate the intersection distance (length) from the reference position point 421 to the intersection point 422. The maximum value of the intersection distance is half the length of the diagonal of the instance bounding box. The intersection distance is divided by half the length of the diagonal of the instance bounding box, thereby realizing the process of normalizing the intersection distance based on the length of the diagonal of the bounding box, and obtaining the sample contour polar diameter in the sample contour polar coordinates corresponding to the intersection point 422.

[0152] Similarly, the normalized sample contour polar diameters corresponding to the intersection points of the multiple rays and the instance contour are obtained through the above normalization method, so that the sample contour polar diameter ranges from 0 to 1.

[0153] Based on the determination of the sample contour polar angle and the sample contour polar diameter, the sample contour polar coordinates corresponding to the multiple included angles are obtained.

[0154] In some embodiments, the distance between the intersection and the pole can also be determined as the sample contour polar diameter; the distance between the intersection and the pole can also be determined, and the ratio of the distance to a preset value can be used as the sample contour polar diameter, etc. This embodiment of the present application is not limited to this.

[0155] Step 330 , predicting the instance contour of the sample instance in the sample image by using the image segmentation model to obtain the predicted instance contour corresponding to the sample instance.

[0156] Among them, the image segmentation model is the model to be trained.

[0157] Step 340 , determining a prediction instance bounding box that surrounds the prediction instance contour from the sample image.

[0158] Optionally, the prediction instance bounding box is implemented as a rectangular shape; or, the prediction instance bounding box is implemented as a circular shape; or, the prediction instance bounding box is implemented as an irregular polygonal shape, etc.

[0159] Illustratively, the prediction instance bounding box is an image region that just surrounds the contour of the prediction instance; or, the prediction instance bounding box is an image region slightly larger than the image region that just surrounds the sample instance, etc.

[0160] Step 350 : Based on the prediction bounding box diagonal of the prediction instance bounding box, determine the center point of the prediction instance bounding box as the prediction reference position point in the prediction instance contour.

[0161] Illustratively, after obtaining the predicted instance bounding box corresponding to the predicted instance bounding box, the predicted bounding box diagonal corresponding to the predicted instance bounding box is determined, and then the center point of the instance bounding box is determined based on at least two predicted bounding box diagonals.

[0162] For example: when the prediction instance bounding box is implemented as a rectangular shape, the two diagonal lines corresponding to the rectangular shape are determined as the diagonal lines of the prediction bounding box, and the intersection of the two prediction bounding box diagonals is determined as the center point of the prediction instance bounding box; or, when the prediction instance bounding box is implemented as a circular shape, at least two diagonal lines (diameters) corresponding to the circular shape are determined as the diagonal lines of the prediction bounding box, and the intersection of at least two bounding box diagonals is determined as the center point of the prediction instance bounding box, etc.

[0163] In some embodiments, the center point of the prediction instance bounding box is used as the prediction reference position point in the prediction instance contour, that is, the prediction reference position point in the sample instance is determined by the position of the prediction instance contour in the sample image.

[0164] Step 360 , by determining the ray distribution law of multiple rays emitted when the polar coordinates of the sample contour are determined, multiple rays are emitted in different directions of the plane where the sample image is located with the predicted reference position point in the predicted instance contour as the pole.

[0165] The ray distribution law is used to characterize the emission of multiple rays emitted when determining the polar coordinates of the sample contour. Schematically, the ray distribution law includes the interval angle between the multiple rays; and / or, the ray distribution law includes the reference position points for emitting the multiple rays; and / or, the ray distribution law includes the emission order of emitting the multiple rays, etc.

[0166] In some embodiments, the reference position point corresponds to the predicted reference position point. Schematically, the reference position point corresponding to the sample instance is implemented as the instance center point of the sample instance; the predicted reference position point corresponding to the predicted instance contour is implemented as the predicted instance contour center point corresponding to the predicted instance contour.

[0167] Schematically, when determining the polar coordinates of the sample contour, the multiple rays emitted are in a manner of emitting one ray in each of the 36 directions of the plane where the sample image is located at a preset interval angle of 10°. When emitting multiple rays with the predicted reference position point in the predicted instance contour as the pole, one ray is emitted in each of the 36 directions of the plane where the sample image is located at a preset interval angle of 10°. The above emission process is regarded as a process of emitting multiple rays in accordance with the ray distribution law, that is, the multiple rays emitted when determining the polar coordinates of the predicted contour comply with the ray distribution law.

[0168] In some embodiments, the predicted reference position point is used as a pole, and rays are emitted in multiple directions of the plane where the sample image is located at preset interval angles.

[0169] The preset interval angle is a preset interval angle, which is used to represent the angle between two adjacent emitted rays. For example, the preset interval angle is 10°, 20°, 15°, etc.

[0170] In some embodiments, when rays are emitted in multiple directions of the plane where the sample image is located at preset interval angles, one ray is emitted in multiple directions of the plane where the sample image is located.

[0171] The multiple rays may be sent simultaneously, sequentially, or at intervals, etc., which is not limited here.

[0172] Step 370, obtaining the polar coordinates of the predicted contour based on the intersection points between the multiple rays and the predicted instance contour.

[0173] Schematically, there are intersections between the multiple rays and the instance contour, and based on the predicted reference position point being the pole, the predicted contour polar coordinates corresponding to the predicted instance contour can be determined.

[0174] In an optional embodiment, a polar axis is determined in the sample image based on the pole, and a polar coordinate system is established.

[0175] In schematic form, after determining the pole corresponding to the predicted reference position point, a ray is randomly selected from multiple rays as the polar axis to establish a polar coordinate system; alternatively, a line segment is randomly determined in the sample image as the polar axis based on the pole, and the line segment may or may not coincide with a ray.

[0176] In an optional embodiment, based on the intersection points between the multiple rays and the predicted instance contour, the polar coordinates of the predicted contour of the multiple intersection points in the polar coordinate system are obtained.

[0177] Illustratively, after obtaining the intersection points between the multiple rays and the predicted instance contour, multiple intersection points are determined, and based on the angles and positions of the multiple intersection points in the polar coordinate system, the predicted contour polar coordinates corresponding to the multiple intersection points are determined.

[0178] The predicted contour polar coordinates are represented by the predicted contour polar angle and the predicted contour polar diameter. The predicted contour polar angle is determined according to the angle between the ray corresponding to the intersection and the polar axis; the predicted contour polar diameter is determined according to the distance between the intersection and the pole.

[0179] Illustratively, based on the position of the intersection point in the polar coordinate system, the angle between the ray corresponding to the intersection point and the polar axis is determined as the predicted contour polar angle in the predicted contour polar coordinates.

[0180] The intersection distance (length) from the predicted reference position point to the intersection point, the maximum value of which is half the length of the diagonal of the predicted bounding box, is divided by half the length of the diagonal of the predicted bounding box of the predicted instance bounding box, thereby realizing the process of normalizing the intersection distance based on the length of the diagonal of the predicted bounding box, and obtaining the predicted contour polar diameter in the predicted contour polar coordinates corresponding to the intersection point.

[0181] In some embodiments, the distance between the intersection and the pole is determined to obtain the intersection distance; the diagonal length of the predicted bounding box corresponding to the predicted instance bounding box is obtained; the intersection distance is normalized based on the diagonal length of the predicted bounding box to obtain the predicted contour polar diameter in the predicted contour polar coordinates.

[0182] Similarly, the normalized predicted contour polar diameters corresponding to the intersection points of the multiple rays and the predicted instance contour are obtained through the above normalization method, so that the range of the predicted contour polar diameter is between 0 and 1.

[0183] Based on the determination of the predicted contour polar angle and the predicted contour polar diameter, the predicted contour polar coordinates corresponding to the multiple angles are obtained.

[0184] In some embodiments, the distance between the intersection and the pole can also be determined as the sample contour polar diameter; the distance between the intersection and the pole can also be determined, and the ratio of the distance to a preset value can be used as the predicted contour polar diameter, etc. This embodiment of the present application is not limited to this.

[0185] Through the above process, there is a corresponding relationship between the predicted contour polar coordinates corresponding to the predicted instance contour and the sample contour polar coordinates, which facilitates the loss value calculation process of the image segmentation model.

[0186] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0187] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in instance contour recognition, avoiding the limitation of the overlap ratio determination method that cannot distinguish sample instances with similar shapes, thereby improving the accuracy and efficiency of instance segmentation.

[0188] In the embodiments of the present application, the contents of determining the polar coordinates of the sample contour and the polar coordinates of the predicted contour are introduced. In the analysis stage of the sample instance, the pole corresponding to the reference position point is determined by the instance bounding box including the sample instance, so that the polar coordinates of the sample contour are determined with the help of multiple rays emitted based on the pole; in addition, when the sample instance is predicted, the predicted instance contour is first roughly obtained, and then the predicted contour polar coordinates are determined based on the intersection between the multiple rays emitted in the analysis stage and the predicted instance contour. Combining the analysis stage and the prediction stage of the sample instance, a more comprehensive analysis of the sample instance can be performed, and the sample contour polar coordinates and the predicted contour polar coordinates determined in the same way are combined, so that the purpose of targeted analysis of the instance contour can be achieved more intuitively through the analysis of the polar coordinates, thereby improving the accuracy of shape analysis.

[0189] In an optional embodiment, the polar coordinates are represented by polar diameters and polar angles, that is, the polar coordinates include polar diameters and polar angles, and when obtaining the loss value based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, the loss value corresponding to the sample instance is obtained by using the sample contour polar diameter and the sample contour polar angle corresponding to the sample contour polar coordinates, and the predicted contour polar diameter and the predicted contour polar angle corresponding to the predicted contour polar coordinates. Schematically, as Figure 5 As shown above Figure 2 The illustrated step 240 may also be implemented as the following steps 510 to 530 .

[0190] Step 510: determine the sample contour polar diameters and sample contour polar angles respectively corresponding to a plurality of sample contour polar coordinates.

[0191] Schematically, after determining multiple sample contour polar coordinates based on the polar coordinate system established in the sample image analysis phase, when it is necessary to determine the loss value based on the sample contour polar coordinates and the predicted contour polar coordinates, the sample contour polar diameters and sample contour polar angles corresponding to the multiple sample contour polar coordinates are obtained. The sample contour polar diameter and the sample contour polar angle are the contents determined when determining the sample contour polar coordinates.

[0192] The method for determining the sample contour polar diameter and the sample contour polar angle has been introduced in the above step 340 and will not be repeated here.

[0193] Step 520: determine the predicted contour polar diameters and predicted contour polar angles corresponding to the plurality of predicted contour polar coordinates.

[0194] Schematically, after determining multiple predicted contour polar coordinates based on the polar coordinate system established in the prediction stage of the sample image, when it is necessary to determine the loss value based on the sample contour polar coordinates and the predicted contour polar coordinates, the predicted contour polar diameters and predicted contour polar angles corresponding to the multiple predicted contour polar coordinates are obtained. The predicted contour polar diameter and the predicted contour polar angle are the contents determined when determining the predicted contour polar coordinates.

[0195] There is a one-to-one correspondence between the multiple predicted contour polar angles and the multiple sample contour polar angles.

[0196] Schematically, when predicting the predicted instance contour, the image segmentation model determines the polar coordinates of the predicted contour in the same way as obtaining the polar coordinates of the sample contour. Therefore, the number, angle and other conditions of the multiple rays emitted in the prediction stage are the same, so that there is a one-to-one correspondence between multiple predicted contour polar angles and multiple sample contour polar angles, which facilitates the training process of the image segmentation model.

[0197] In some embodiments, the process of determining the predicted contour polar coordinates in the manner of obtaining the sample contour polar coordinates is implemented by adjusting the model parameters of the image segmentation model.

[0198] The method for determining the predicted contour polar diameter and the predicted contour polar angle has been introduced in the above step 360 and will not be repeated here.

[0199] Step 530 , obtaining a loss value corresponding to the sample instance based on the difference between the sample contour polar diameter and the predicted contour polar diameter under the corresponding relationship between the multiple predicted contour polar angles and the multiple sample contour polar angles.

[0200] Schematically, based on the one-to-one correspondence between multiple predicted contour polar angles and multiple sample contour polar angles, the difference between polar angles can be compared at the same angle.

[0201] In an optional embodiment, for a first predicted contour polar coordinate in a plurality of predicted contour polar coordinates, a first predicted contour polar angle in the first predicted contour polar coordinate is determined.

[0202] The first predicted contour polar coordinate is any one of the multiple predicted contour polar coordinates, and is referred to herein as the first predicted contour polar coordinate to determine the first predicted contour polar angle in the first predicted contour polar coordinate.

[0203] In an optional embodiment, a first sample contour polar angle having the same angle as the first predicted contour polar angle is determined from the sample contour polar angles respectively corresponding to a plurality of sample contour polar coordinates.

[0204] Illustratively, the first predicted contour polar angle is 30 degrees, and a first sample contour polar angle of 30 degrees is determined from the sample contour polar angles corresponding to the plurality of sample contour polar coordinates.

[0205] In an optional embodiment, a first predicted contour polar diameter corresponding to the first predicted contour polar angle is determined, and a first sample contour polar diameter corresponding to the first sample contour polar angle is determined.

[0206] Illustratively, based on the first predicted contour polar coordinates and the first predicted contour polar angle, a first predicted contour polar diameter in the first predicted contour polar coordinates is determined.

[0207] Similarly, based on the first sample contour polar coordinates and the first sample contour polar angle, the first sample contour polar diameter in the first sample contour polar coordinates is determined.

[0208] In an optional embodiment, a loss value corresponding to the sample instance is obtained based on the difference between the first predicted contour polar diameter and the first sample contour polar diameter.

[0209] Schematically, the first predicted contour polar diameter and the first sample contour polar diameter are determined based on the correspondence between the predicted contour polar angle and the sample contour polar angle, so the difference between the first predicted contour polar diameter and the first sample contour polar diameter corresponds to the first predicted contour polar angle, and the first sample contour polar angle and the first predicted contour polar angle have the same angle, so the difference between the first predicted contour polar diameter and the first sample contour polar diameter can be used as the loss corresponding to the angle.

[0210] Optionally, the loss value corresponding to the sample instance is determined by synthesizing the losses corresponding to the multiple angles, wherein the multiple angles are used to represent the angle between two adjacent rays.

[0211] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0212] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in identifying instance contours, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation.

[0213] In the embodiment of the present application, the loss value content corresponding to the sample instance is determined based on the relationship between the polar diameter and the polar angle. Since the rays emitted when determining the polar coordinates of the predicted contour and the rays emitted when determining the polar coordinates of the sample contour are in a one-to-one correspondence, there is a one-to-one correspondence between the polar angle of the predicted contour in the polar coordinates and the polar angle of the sample contour in the polar coordinates of the sample contour; under the condition of using the corresponding relationship of the polar angles in the polar coordinates, the difference between the polar diameter of the predicted contour in the polar coordinates of the predicted contour and the polar diameter of the sample contour in the polar coordinates of the sample contour is compared, so that the loss value corresponding to the sample instance can be obtained more quickly and accurately, avoiding the problem of low analysis efficiency and poor analysis accuracy caused by directly comparing the similarity between the predicted instance contour and the instance contour, and improving the accuracy of determining the loss value with the help of further analysis of the predicted instance contour and the instance contour.

[0214] In an optional embodiment, the deviation determined by the difference between the polar coordinates of the sample contour and the polar coordinates of the predicted contour is called a contour deviation value, and the loss value corresponding to the sample instance is not only related to the contour deviation value, but also to the coordinate deviation value corresponding to the sample instance, and the coordinate deviation value is used to describe the offset of the position. Schematically, as Figure 6 As shown above Figure 2 The illustrated step 240 may also be implemented as the following steps 610 to 650 .

[0215] Step 610 , obtaining a contour deviation value based on a contour difference between the sample contour polar coordinates and the predicted contour polar coordinates for describing the shape.

[0216] Schematically, the loss determined by the difference between the polar coordinates of the sample contour and the polar coordinates of the predicted contour is called the contour deviation value. The contour deviation value is used to describe the contour difference between the instance contour corresponding to the sample instance and the predicted instance contour predicted by the image segmentation model, and characterizes the shape prediction corresponding to the sample instance.

[0217] Optionally, the contour deviation value between the sample contour polar coordinates and the predicted contour polar coordinates is already in the above Figure 5 The above has been described in the illustrated embodiment and will not be described in detail here.

[0218] Step 620: perform feature compression on the sample image through the image segmentation model to obtain a sample feature map.

[0219] Schematically, the model structure of the image segmentation model is pre-constructed so that in addition to outputting the prediction results of the sample instance (such as the above-mentioned predicted contour polar coordinates), the model is also used to perform convolution and pooling processing on the sample image through convolution layers and pooling layers, and the downsampling processing such as convolution and pooling is used as a feature compression method to obtain a sample feature map corresponding to the sample image.

[0220] Among them, the sample feature map is composed of multiple sample feature grids.

[0221] Schematically, different sample feature grids are used to represent information of different image regions in the sample image.

[0222] Optionally, after the sample image is feature compressed using different sampling multiples, a sample feature map with different grid sizes will be obtained, and the grid size is used to represent the size of the sample feature grid.

[0223] Schematically, multiple sample feature grids under the same sample feature map have the same grid size.

[0224] Step 630 : determining a first sample feature grid including the sample instance from a plurality of sample feature grids based on the instance position of the sample instance in the sample image.

[0225] The instance position is used to represent the position information of the sample instance in the sample image. For example, the instance position is implemented as the pixel position coordinates corresponding to multiple pixel points in the sample instance; or the instance position is implemented as the instance center point coordinates corresponding to the instance center point in the sample instance.

[0226] In some embodiments, based on the instance position of the sample instance in the sample image, a first sample feature grid including the sample instance is determined from a plurality of sample feature grids.

[0227] Among them, including a grid used to represent that the first sample feature includes all or part of the sample instances.

[0228] In an optional embodiment, based on the instance position of the sample instance in the sample image, an instance bounding box surrounding the sample instance is determined; based on the bounding box diagonal of the instance bounding box, a center point of the instance bounding box is determined as the instance center point of the sample instance.

[0229] Schematically, the instance center point is the center point position of the sample instance determined by means of the instance bounding box.

[0230] In an optional embodiment, a sample feature grid where the instance center point is located is determined from a plurality of sample feature grids as the sample feature grid.

[0231] Schematically, multiple sample feature grids have definite positions in the sample feature map, and the instance center point determined based on the instance position also has a definite position, so that the sample feature grid where the instance center point is located can be determined from the multiple sample feature grids and used as the sample feature grid.

[0232] In an optional embodiment, the area intersection between the sample feature grid and the sample instance is determined, and the area union between the sample feature grid and the sample instance is determined.

[0233] Among them, the region intersection is used to represent the intersection between the region corresponding to the sample feature grid and the region corresponding to the sample instance; the region union is used to represent the union between the region corresponding to the sample feature grid and the region corresponding to the sample instance.

[0234] Optionally, since the sample feature grid is the sample feature grid within the sample feature map after feature compression, when determining the region intersection and region union, the sample feature map can be restored to determine the restored sample feature grid, thereby determining the region intersection and region union with the help of the region relationship between the restored sample feature grid and the sample instance; or, the sample instance can be processed according to the sampling ratio corresponding to the feature compression to obtain the processed sample instance, thereby determining the region intersection and region union with the help of the region relationship between the processed sample instance and the sample feature grid.

[0235] In an optional embodiment, in response to the ratio between the region intersection and the region union reaching a preset threshold, the sample feature grid is used as the first sample feature grid including the sample instance.

[0236] Illustratively, the preset threshold is a predetermined threshold, for example, the preset threshold is a value such as 0.5, 0.6, or the preset threshold is a range of 0.4 to 0.8.

[0237] Optionally, the ratio between the region intersection and the region union is determined, called the intersection-and-union ratio, and then the intersection-and-union ratio is compared with a preset threshold. When the intersection-and-union ratio reaches the preset threshold, the sample feature grid is used as the first sample feature grid, thereby determining that the sample instance is included in the first sample feature grid.

[0238] That is, the relationship is implemented as a relationship in which the center point of the instance is in the first sample feature grid and the intersection and union ratio reaches a preset ratio.

[0239] In some embodiments, in response to the ratio between the region intersection and the region union not reaching a preset threshold, the sample instance is analyzed using a sample feature map determined by other sampling multiples to determine a first sample feature grid including the sample instance.

[0240] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0241] Step 640: Determine a coordinate deviation value based on the position offset relationship between the sample instance and the first sample feature grid.

[0242] Illustratively, after the sample instance and the first sample feature grid including the sample instance are determined, the position offset between the sample instance and the first sample feature grid is analyzed.

[0243] In an optional embodiment, the position offset relationship between the sample instance and the first sample feature grid is determined based on the instance position to obtain coordinate offset data.

[0244] The coordinate offset data is used to represent the position offset between the sample instance and the first sample feature grid in the data form of the coordinate data.

[0245] In some embodiments, based on the instance position, an instance bounding box surrounding the sample instance is determined from the sample image; based on the bounding box diagonal of the instance bounding box, a center point of the instance bounding box is determined as the instance center point of the sample instance.

[0246] Schematically, the instance center point is the center point position of the sample instance determined by means of the instance bounding box.

[0247] In some embodiments, a first grid center point corresponding to the first sample feature grid is determined.

[0248] Illustratively, the first sample feature grid is implemented as a rectangular area, the center point of the first sample feature grid is determined by a diagonal line within the first sample feature grid, and the center point is used as the first grid center point corresponding to the first sample feature grid.

[0249] In some embodiments, the coordinate offset data is obtained based on the position offset relationship between the instance center point and the first grid center point.

[0250] Schematically, after obtaining the instance center point, the instance coordinate data of the instance center point in a two-dimensional coordinate system is determined, and the first grid coordinate data of the first grid center point in the same two-dimensional coordinate system is determined, thereby determining the position offset of the instance center point relative to the first grid center point based on the coordinate offset between the instance coordinate data and the first grid coordinate data, and obtaining the coordinate offset data.

[0251] The coordinate offset data is used to characterize the position offset of the sample instance relative to the first sample feature grid.

[0252] In an optional embodiment, the predicted instance position of the sample instance in the sample image is predicted by an image segmentation model.

[0253] Schematically, the image segmentation model has a certain image segmentation capability and can roughly determine the predicted instance position of the sample instance in the sample image. The predicted instance position is used to characterize the predicted position of the sample instance in the sample image, and roughly illustrates the position of the sample instance predicted by the image segmentation model.

[0254] In an optional embodiment, a second sample feature grid is obtained from a plurality of sample feature grids based on the predicted instance position.

[0255] Illustratively, in addition to multiple sample feature grids participating in determining coordinate offset data, when a sample instance in a sample image is predicted by an image segmentation model, a second sample feature grid is also obtained from the multiple sample feature grids based on the predicted instance position.

[0256] Optionally, a sample feature grid having an inclusion relationship with the predicted instance position is determined from the plurality of sample feature grids as the second sample feature grid, wherein the inclusion relationship can be implemented as the relationship between the above-mentioned sample instance and the first sample feature grid.

[0257] The second sample feature grid is a sample feature grid determined based on the prediction of the position of the sample instance, so the second sample feature grid and the first sample feature grid may be the same or different. When the second sample feature grid and the first sample feature grid are the same, it indirectly proves that the prediction accuracy of the image segmentation model is high; when the second sample feature grid and the first sample feature grid are different, it indirectly proves that the prediction accuracy of the image segmentation model is low.

[0258] In some embodiments, based on the predicted instance position, a predicted instance bounding box surrounding the sample instance is determined; based on the predicted instance bounding box diagonal of the predicted instance bounding box, a center point of the predicted instance bounding box is determined as the predicted instance center point of the sample instance.

[0259] A sample feature grid where the center point of the predicted instance is located is determined from multiple sample feature grids, and the intersection-and-union ratio between the predicted instance position and the sample feature grid reaches a preset threshold, and the sample feature grid is used as the second sample feature grid.

[0260] In an optional embodiment, prediction offset data is obtained based on a position offset relationship between the predicted instance position and the second sample feature grid.

[0261] Optionally, a second grid center point corresponding to the second sample feature grid is determined; and based on a position offset relationship between the predicted instance center point and the second grid center point, predicted offset data is obtained.

[0262] The coordinate offset data is used to characterize the predicted position offset of the predicted instance position relative to the second sample feature grid.

[0263] That is: when analyzing the position of the sample instance in the sample image, the position offset relationship between the sample instance (or instance position) and the corresponding first sample feature grid will be determined to obtain coordinate offset data; the position offset relationship between the predicted instance position and the corresponding second sample feature grid will also be determined to obtain predicted offset data.

[0264] In an optional embodiment, the coordinate deviation value is obtained through the coordinate offset data and the predicted offset data.

[0265] Illustratively, the coordinate offset data and the predicted offset data are both based on differences determined by the coordinates.

[0266] The coordinate offset data is a value determined based on the instance coordinate data of the instance center point and the first grid coordinate data of the first sample feature grid. For example, if the instance coordinate data of the instance center point is (1, 2), and the first grid coordinate data of the first grid center point of the first sample feature grid is (1.1, 1.8), then the coordinate offset data is approximately 0.22≈0.1 2 +0.2 2 ; Or, the first coordinate offset data in the coordinate offset data representing the horizontal axis offset is 0.1=(1.1-1), and the second coordinate offset data in the coordinate offset data representing the vertical axis offset is -0.2=(1.8-2).

[0267] Similarly, the predicted offset data is a value determined based on the predicted instance coordinate data of the predicted instance center point and the second grid coordinate data of the second sample feature grid. For example, if the instance coordinate data of the predicted instance center point is (1.1, 2), and the second grid coordinate data of the second sample feature grid is (1.2, 1.9), then the predicted offset data is approximately 0.22≈0.1 2 +0.2 2 ; Or, the first predicted offset data representing the horizontal axis offset in the predicted offset data is 0.1=(1.2-1.1), and the second predicted offset data representing the vertical axis offset in the predicted offset data is -0.1=(1.9-2).

[0268] In order to facilitate obtaining the coordinate deviation value through the coordinate offset data and the predicted offset data, the coordinate offset data and the predicted offset data are expressed in the same form.

[0269] Optionally, both the coordinate offset data and the predicted offset data are determined by combining the horizontal axis offset situation and the vertical axis offset situation. When comparing the loss between the coordinate offset data and the predicted offset data, the first coordinate offset data representing the horizontal axis offset situation in the coordinate offset data is determined, and the first predicted offset data representing the horizontal axis offset situation in the predicted offset data is determined, and the horizontal axis offset situation between the coordinate offset data and the predicted offset data is determined by combining the first coordinate offset data and the first predicted offset data; similarly, the second coordinate offset data representing the vertical axis offset situation in the coordinate offset data is determined, and the second predicted offset data representing the vertical axis offset situation in the predicted offset data is determined, and the vertical axis offset situation between the coordinate offset data and the predicted offset data is determined by combining the second coordinate offset data and the second predicted offset data.

[0270] In an illustrative manner, the coordinate deviation value is determined by combining the horizontal axis offset and the vertical axis offset.

[0271] Step 650, based on the coordinate deviation value and the contour deviation value, obtain the loss value corresponding to the sample instance.

[0272] Schematically, after obtaining the coordinate deviation value representing the position and the contour deviation value representing the shape, the coordinate deviation value and the contour deviation value are integrated to obtain the loss value corresponding to the sample instance.

[0273] Optionally, coordinate deviation values ​​and contour deviation values ​​respectively corresponding to a plurality of sample instances are determined, and then the sum of the plurality of coordinate deviation values ​​and the plurality of contour deviation values ​​is combined to obtain a loss value corresponding to the sample instance.

[0274] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0275] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in identifying instance contours, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation.

[0276] In the embodiment of the present application, a process of determining the loss value by synthesizing the contour deviation value and the coordinate deviation value determined by polar coordinates is introduced. The sample image is feature compressed by an image segmentation model to obtain a sample feature map, and then based on the instance position of the sample instance in the sample image, a first sample feature grid including the sample instance is determined from multiple sample feature grids, and the coordinate deviation value is obtained by analyzing the position offset relationship between the sample instance and the first sample feature grid, and by predicting the position offset relationship between the instance contour and the second sample feature grid in the prediction stage. By synthesizing the coordinate offset data in the analysis stage and the predicted offset data in the prediction stage, a more comprehensive analysis process of the sample instance is achieved. With the help of the introduced sample feature map, the problem of low analysis accuracy caused by directly comparing the distance difference between the instance center point and the predicted instance center point can be avoided. With the help of the deviation between the instance center point and the first grid center point, and the deviation between the predicted instance center point and the first grid center point, the deviations of the two are combined to improve the analysis accuracy of the instance position, thereby improving the accuracy of determining the loss value.

[0277] In an optional embodiment, the loss value corresponding to the sample instance is related to the size deviation value used to describe the size of the sample instance in addition to the contour deviation value and the coordinate deviation value. Figure 7 As shown above Figure 2 The illustrated step 240 may also be implemented as the following steps 710 to 740 .

[0278] Step 710 : Determine instance size data corresponding to the sample instance based on the instance region of the sample instance in the sample image.

[0279] The instance size data is used to characterize the instance size of the sample instance in the sample image, that is, the instance size data is data for measuring the size of the sample instance.

[0280] In some embodiments, the diagonal length corresponding to the sample instance is used as the instance size data. For example, when the shape of the sample instance is a regular shape, the instance size data corresponding to the sample instance is determined using the diagonal calculation formula corresponding to the regular shape; or, when the shape of the sample instance is an irregular shape, the instance size data corresponding to the sample instance is determined using the instance bounding box surrounding the sample instance.

[0281] Optionally, the area corresponding to the sample instance is used as the instance size data; or, the perimeter corresponding to the sample instance is used as the instance size data, etc. The above are merely illustrative examples and are not limited to the embodiments of the present application.

[0282] In an optional embodiment, the diagonal length of the grid corresponding to the first sample feature grid is determined.

[0283] Illustratively, the first sample feature grid is a sample feature grid determined from a plurality of sample feature grids in the sample feature map and includes sample instances.

[0284] Optionally, multiple sample feature grids in the sample feature map have the same grid size, such as: the areas of the multiple sample feature grids are the same, the grid lengths of the multiple sample feature grids are the same, and the grid widths of the multiple sample feature grids are the same (therefore, the grid diagonal lengths of the multiple sample feature grids are also the same), etc.

[0285] Illustratively, the grid length and grid width corresponding to the first sample feature grid are determined to obtain the grid diagonal length (the grid diagonal length corresponding to the first sample feature grid may also be referred to as the first grid diagonal length).

[0286] In some embodiments, the sample feature map is considered to be obtained after feature compression of the sample image. Therefore, when determining the grid diagonal length, the sample feature map is enlarged to the image size corresponding to the sample image, so as to determine the enlarged first sample feature grid and the grid diagonal length of the enlarged first sample feature grid, and the grid diagonal length is used as the above-mentioned grid diagonal length corresponding to the first sample feature grid.

[0287] In an optional embodiment, an instance bounding box surrounding the sample instance is determined based on the instance position; based on the bounding box diagonal length corresponding to the instance bounding box, the quotient of the bounding box diagonal length and the grid diagonal length is determined to obtain instance size data corresponding to the sample instance.

[0288] In an illustrative manner, an instance bounding box surrounding the sample instance is determined, and the diagonal length of the bounding box corresponding to the instance bounding box is determined. For example, the instance bounding box is implemented as a rectangle, and the diagonal length of the bounding box corresponding to the instance bounding box is determined based on the length and width of the instance bounding box; or, the instance bounding box is implemented as a circle, and the diagonal length of the bounding box corresponding to the instance bounding box is the diameter of the instance bounding box, etc.

[0289] Optionally, the diagonal length of the bounding box is divided by the diagonal length of the grid, and the quotient of the lengths is used as instance size data corresponding to the sample instance.

[0290] Step 720: predict the instance size corresponding to the sample instance through the image segmentation model to obtain predicted size data.

[0291] Illustratively, a predicted instance contour corresponding to the sample instance is predicted by an image segmentation model, and a second sample feature grid corresponding to the sample instance is determined from multiple sample feature grids based on the predicted instance contour; and a second grid diagonal length corresponding to the second sample feature grid is determined.

[0292] In some embodiments, the sample feature map is considered to be obtained after feature compression of the sample image. Therefore, when determining the second grid diagonal length, the sample feature map is enlarged to the image size corresponding to the sample image, so as to determine the enlarged second sample feature grid and the grid diagonal length of the enlarged second sample feature grid, and the grid diagonal length is used as the second grid diagonal length corresponding to the second sample feature grid.

[0293] In an optional embodiment, a predicted instance bounding box surrounding the sample instance is obtained based on the predicted instance contour prediction; based on the predicted bounding box diagonal length corresponding to the predicted instance bounding box, the quotient of the predicted bounding box diagonal length and the second grid diagonal length is determined to obtain predicted size data output by the image segmentation model.

[0294] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0295] Step 730: Obtain a size deviation value based on the instance size data and the predicted size data.

[0296] Illustratively, the size deviation value is obtained based on the difference between the instance size data and the predicted size data.

[0297] Step 740, based on the size deviation value, the coordinate deviation value and the contour deviation value, obtain the loss value corresponding to the sample instance.

[0298] In an illustrative manner, the size deviation value, coordinate deviation value, and contour deviation value corresponding to the sample instance are integrated to obtain the loss value corresponding to the sample instance.

[0299] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0300] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in identifying instance contours, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation.

[0301] In the embodiments of the present application, a process of determining a loss value by integrating contour deviation values, coordinate deviation values, and size loss values ​​is introduced. Based on the image area of ​​the sample instance in the sample image, the instance size data of the sample instance is determined, and the instance size corresponding to the sample instance is predicted by the image segmentation model to obtain predicted size data, thereby obtaining a size deviation value. The process of determining the instance size data and the predicted size data introduces a first sample feature grid, so the determination process can achieve the purpose of more accurate size analysis with the help of the first sample feature grid with a relatively clear grid size, thereby improving the prediction accuracy of the size deviation value and improving the accuracy of obtaining the loss value.

[0302] In an optional embodiment, when analyzing the sample image with the aid of the image segmentation model, the sample image is subjected to feature compression at different sampling multiples through the image segmentation model, and sample feature maps corresponding to different sampling multiples are obtained. Figure 8 As shown above Figure 6 The step 620 shown can also be implemented as the following steps 810 to 820; Figure 6 The shown step 630 can also be implemented as the following steps 830 to 863; the above process of obtaining the loss value corresponding to the sample instance can also be determined based on the sample feature maps corresponding to different sampling multiples, and implemented as the following steps 870 to 890.

[0303] Step 810: Perform feature compression on the sample image at a first sampling multiple using an image segmentation model to obtain a first sample feature map.

[0304] The first sample feature map includes a plurality of sample feature grids having a first grid size, and the first grid size is related to the first sampling multiple.

[0305] Optionally, in addition to being related to the first sampling multiple, the first grid size is also related to an image size corresponding to the sample image.

[0306] Step 820: Perform feature compression on the sample image at a second sampling multiple using an image segmentation model to obtain a second sample feature map.

[0307] Among them, the second sample feature map includes multiple sample feature grids with a second grid size, the second grid size is related to the second sampling multiple, and the second sampling multiple is different from the first sampling multiple, and the second grid size is different from the first grid size.

[0308] Similarly, the second grid size is not only related to the second sampling multiple, but also related to the image size corresponding to the sample image.

[0309] Schematically, taking sample image A as an example, the image segmentation model is used to perform feature compression on sample image A at a first sampling multiple to obtain a first sample feature map. In addition, the image segmentation model is used to perform feature compression on sample image A at a second sampling multiple to obtain a second sample feature map.

[0310] In an optional embodiment, an image segmentation model is trained using multiple sample images, and for each sample image in the multiple sample images, the image segmentation model performs feature compression on the sample image using the first sampling multiple and the second sampling multiple, respectively, to obtain a first sample feature map and a second sample feature map.

[0311] It is worth noting that the above-mentioned first sampling multiple and second sampling multiple are only illustrative examples, and the sample image can also be subjected to feature compression processing by other sampling multiples to obtain a corresponding sample feature map, which is not limited in this embodiment of the present application.

[0312] In some embodiments, the sample image is processed with a first sampling multiple, a second sampling multiple, and a third sampling multiple, and three sample feature maps with different feature map sizes are obtained, wherein the grid sizes of the three sample feature maps are also different, such as Fig. 9 As shown, this is a schematic diagram of the model structure of the image segmentation model.

[0313] Schematically, to facilitate the image segmentation model to process the image, the image scaled to a certain image size is called a sample image, that is, the image sizes of multiple sample images used to train the image segmentation model are consistent. The following description is taken as an example of analyzing the sample image 910 through the image segmentation model.

[0314] Considering the model complexity and the speed of instance segmentation, an image segmentation model is constructed through a small number of convolutional layers and pooling layers, and the sample image 910 is scaled to 1000*600 pixels (image size).

[0315] Optionally, feature compression is performed on the sample image 910 by using 3 sampling multiples, such as: first, the sample image 910 is passed through the convolution layer 921, where the size of the convolution kernel is 3, the step size is 1, and the number of output channels is 16 (which can be understood as the number of convolution kernels is 16). Based on the image size of the sample image 910 being 1000*600, the output of the convolution layer 921 is determined to be 998*598*16, where 998=(1000-3) / 1+1; 598=(600-3) / 1+1; similarly, the output 998*598*16 of the convolution layer 921 is input into the convolution layer 922, where the size of the convolution kernel is 3, the step size is 1, and the number of output channels is 16. The length is 2, and the number of output channels is 32 (which can be understood as the number of convolution kernels is 32). Then the output of the convolution layer 922 is determined to be 499*299*32, where 499=(998-3) / 2+1; 299=(598-3) / 2+1; the output of the convolution layer 922 499*299*32 is input to the pooling layer 923, where the pooling layer is 2 (2*2), then the output of the pooling layer 923 is determined to be 250*150*32, where there is a padding process, such as: the padding value is 1, then 250=(499+1) / 2; 150=(299+1) / 2. And so on for the output of each layer in the image segmentation model.

[0316] Among them, the first sampling multiple is 32 times, and the feature map size of the first sample feature map is 30*18, that is: the output of the convolution layer 931; the second sampling multiple is 64 times, and the feature map size of the second sample feature map is 15*9, that is: the output of the convolution layer 932; the third sampling multiple is 128 times, and the feature map size of the third sample feature map is 7*4, that is: the output of the convolution layer 933.

[0317] Among them, when the image segmentation model is trained, the model structure of the obtained image segmentation model is consistent with the model structure of the image segmentation model, wherein the parameters of the convolution layer and / or pooling layer may have been adjusted. The above is only an illustrative example and the embodiments of the present application are not limited to this.

[0318] Step 830 : Determine an instance bounding box surrounding the sample instance based on the instance position of the sample instance in the sample image.

[0319] Illustratively, according to the instance position of the sample instance in the sample image, a region that just surrounds the sample instance image is determined as the instance bounding box.

[0320] Step 840 : Based on the diagonal of the instance bounding box, determine the center point of the instance bounding box as the instance center point of the sample instance.

[0321] Schematically, the instance bounding box is implemented as a rectangular shape, and the bounding box length and bounding box width corresponding to the instance bounding box are determined to obtain the bounding box diagonal. The center point of the instance bounding box is obtained by combining the intersection of the two bounding box diagonals, and the center point is used as the instance center point of the sample instance.

[0322] Step 851: Determine, from a plurality of sample feature grids having a first grid size, a sample feature grid where an instance center point is located as a first instance feature grid.

[0323] Schematically, taking the process of performing feature compression processing on a sample image with two sampling multiples as an example, based on the position of the sample instance and the position of the determined instance center point, the sample feature grid where the instance center point is located can be determined from multiple sample feature grids with a first grid size in the first sample feature map, and the sample feature grid is used as the first instance feature grid, and the grid size of the first instance feature grid is the first grid size.

[0324] like Fig.10 As shown, it is a schematic diagram of the first sample feature map. Based on the diagonal of the instance bounding box surrounding the sample instance 1010, the center point of the instance bounding box is determined as the instance center point 1011 of the sample instance, thereby determining the sample feature grid where the instance center point is located from multiple sample feature grids with the first grid size in the first sample feature map as the first instance feature grid 1020.

[0325] Step 852 , determining a first region intersection between the first instance feature grid and the sample instance, and determining a first region union between the first instance feature grid and the sample instance.

[0326] Illustratively, the first region intersection is used to represent the intersection between the region corresponding to the first instance feature grid and the region corresponding to the sample instance; the first region union is used to represent the union between the region corresponding to the first instance feature grid and the region corresponding to the sample instance.

[0327] Step 853 , in response to the ratio between the first region intersection and the first region union reaching a preset threshold, taking the first instance feature grid as a first sample feature grid including the sample instance.

[0328] Schematically, the ratio obtained by dividing the intersection of the first regions and the union of the first regions is called a first intersection-and-union ratio. The first intersection-and-union ratio is compared with a preset threshold value. When the first intersection-and-union ratio reaches the preset threshold value, the first instance feature grid is used as the first sample feature grid including the sample instance.

[0329] For example, the ratio between the intersection of the first regions and the union of the first regions is equal to a preset threshold; or the ratio between the intersection of the first regions and the union of the first regions is greater than the preset threshold.

[0330] Illustratively, the preset threshold is 0.5, and when the ratio between the first region intersection and the first region union is greater than or equal to the preset threshold, the first instance feature grid is taken as the first sample feature grid including the sample instance.

[0331] Step 861: Determine, from a plurality of sample feature grids having a second grid size, a sample feature grid where an instance center point is located as a second instance feature grid.

[0332] Schematically, taking the process of performing feature compression processing on the sample image with two sampling multiples as an example, based on the position of the sample instance and the position of the determined instance center point, the sample feature grid where the instance center point is located can be determined from multiple sample feature grids with a second grid size in the second sample feature map, and the sample feature grid is used as the second instance feature grid, and the grid size of the second instance feature grid is the second grid size.

[0333] like Fig.11 As shown, it is a schematic diagram of the second sample feature map. Based on the diagonal of the instance bounding box surrounding the sample instance 1110, the center point of the instance bounding box is determined as the instance center point 1111 of the sample instance, thereby determining the sample feature grid where the instance center point is located from multiple sample feature grids with a second grid size in the second sample feature map as the second instance feature grid 1120.

[0334] Step 862, determining a second region intersection between the second instance feature grid and the sample instance, and determining a second region union between the second instance feature grid and the sample instance.

[0335] Illustratively, the second region intersection is used to represent the intersection between the region corresponding to the second instance feature grid and the region corresponding to the sample instance; the second region union is used to represent the union between the region corresponding to the second instance feature grid and the region corresponding to the sample instance.

[0336] Step 863 , in response to the ratio between the second region intersection and the second region union reaching a preset threshold, taking the second instance feature grid as the first sample feature grid including the sample instance.

[0337] Schematically, the ratio obtained by dividing the intersection of the second area and the union of the second area is called the second intersection-and-union ratio. The second intersection-and-union ratio is compared with a preset threshold. When the second intersection-and-union ratio reaches the preset threshold, the second instance feature grid is used as the first sample feature grid including the sample instance.

[0338] The preset threshold value compared with the second intersection-and-union ratio may be the same as or different from the preset threshold value compared with the first intersection-and-union ratio, which is not limited here.

[0339] Illustratively, the preset threshold is 0.5, and when the ratio between the second region intersection and the second region union is greater than or equal to the preset threshold, the second instance feature grid is used as the first sample feature grid including the sample instance.

[0340] In an optional embodiment, based on instance positions of the multiple sample instances respectively corresponding to the multiple sample instances in the sample image, a first sample feature grid corresponding to each sample instance is determined from the multiple sample feature grids.

[0341] Illustratively, if a sample image includes multiple sample instances, a first sample feature grid corresponding to each sample instance is determined from multiple sample feature grids. Due to different sampling multiples, the sample feature grids where the instance center points of the sample instances are located may be different, resulting in different intersection-and-union ratios between the sample instances and the corresponding sample feature grids. Therefore, the first sample feature grids corresponding to different sample instances may be located in different feature maps.

[0342] For example: the sample image includes sample instance A and sample instance B. The first sample feature grid corresponding to sample instance A is located in the first sample feature map, that is, the grid size of the first sample feature grid corresponding to sample instance A is the first grid size; the first sample feature grid corresponding to sample instance B is located in the second sample feature map, that is, the grid size of the second sample feature grid corresponding to sample instance B is the second grid size, and so on.

[0343] In some embodiments, in response to the ratio between the intersection of the first area and the union of the first area not reaching a preset threshold, steps 861 to 863 as above are executed; in response to the ratio between the intersection of the second area and the union of the second area not reaching the preset threshold, steps 851 to 853 as above are executed; in response to the ratio between the intersection of the first area and the union of the first area not reaching the preset threshold, and in response to the ratio between the intersection of the second area and the union of the second area not reaching the preset threshold, similar analysis as above is performed on the sample feature map after feature compression using other sampling multiples, which will not be repeated here.

[0344] That is, it should be noted that steps 851 to 853 and steps 861 to 863 can be implemented as simultaneous execution or sequential execution. For example, first execute steps 851 to 853, and if the ratio between the intersection of the first region and the union of the first region does not reach the preset threshold, execute steps 861 to 863; or, first execute steps 861 to 863, and if the ratio between the intersection of the second region and the union of the second region does not reach the preset threshold, execute steps 851 to 853. The embodiments of the present application do not limit this.

[0345] Step 870, in response to the first sample feature grid being a sample feature grid with a first grid size, obtaining a first loss value corresponding to the sample instance under the first sample feature map based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates.

[0346] Illustratively, multiple sample instances in a sample image are analyzed, or at least one sample instance respectively included in multiple samples is analyzed, thereby realizing the process of analyzing multiple sample instances.

[0347] For any sample instance among multiple sample instances, after determining a first sample feature grid corresponding to the sample instance, if the first sample feature grid is a sample feature grid with a first grid size, based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtain a first loss value corresponding to the sample instance under the first sample feature map.

[0348] Similarly, if the first sample feature grids corresponding to some sample instances in multiple sample implementation embodiments are sample feature grids with a first grid size, then based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, the contour deviation values ​​corresponding to the some sample instances are determined, and the first loss value corresponding to the sample instance under the first sample feature graph is obtained by combining the contour deviation values ​​corresponding to the some sample instances.

[0349] In some embodiments, when obtaining the first loss value, the contour deviation values, coordinate deviation values, and size deviation values ​​corresponding to some sample instances are comprehensively considered to determine the first loss value.

[0350] Step 880, in response to the first sample feature grid being a sample feature grid with a second grid size, based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtain a second loss value corresponding to the sample instance under the second sample feature map.

[0351] Illustratively, for any one sample instance among multiple sample instances, after determining the first sample feature grid corresponding to the sample instance, if the first sample feature grid is a sample feature grid with a second grid size, based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtain a second loss value corresponding to the sample instance under the second sample feature map.

[0352] Similarly, if the first sample feature grids corresponding to another part of the sample instances in multiple sample embodiments are sample feature grids with a second grid size, then based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, the contour deviation values ​​corresponding to the other part of the sample instances are determined, and the contour deviation values ​​corresponding to the comprehensive part of the sample instances are used to obtain the second loss value corresponding to the sample instances under the second sample feature graph.

[0353] In some embodiments, when obtaining the second loss value, the contour deviation values, coordinate deviation values, and size deviation values ​​corresponding to another portion of sample instances are combined to determine the second loss value.

[0354] Step 890: Taking the sum of the first loss value and the second loss value as the loss value corresponding to the sample instance.

[0355] Schematically, after obtaining a first loss value corresponding to the first sample feature map and a second loss value corresponding to the second sample feature map, the sum of the first loss value and the second loss value is calculated to obtain a loss value corresponding to the sample instance, which is obtained based on multiple sample instances and is related to multiple sample instances.

[0356] In an optional embodiment, the process of determining the loss value is to analyze the sample feature graphs corresponding to three sampling multiples, for example, Fig. 9 As shown, the first sample feature map output by the convolution layer 931, the second sample feature map output by the convolution layer 932, and the third sample feature map output by the convolution layer 933, wherein the number of output channels is 40, and the 40 channels include the following contents:

[0357] Channel 1: The horizontal axis offset data of the instance center point and the grid center point of the corresponding sample feature grid for the horizontal axis offset situation.

[0358] Wherein, in the analysis stage (analyzing the sample image marked with the polar coordinates of the sample contour): the horizontal axis offset data is implemented as the first coordinate offset data of the instance center point of the sample instance and the first grid center point of the first sample feature grid for the horizontal axis offset situation;

[0359] In the prediction stage (prediction is performed on the above sample image ignoring the polar coordinates of the sample contour): the horizontal axis offset data is implemented as the first prediction offset data for the horizontal axis offset situation between the predicted instance center point of the sample instance and the second grid center point of the second sample feature grid.

[0360] Channel 2: The longitudinal axis offset data of the instance center point and the grid center point of the corresponding sample feature grid for the longitudinal axis offset condition.

[0361] Wherein, in the analysis stage: the longitudinal axis offset data is implemented as second coordinate offset data of the instance center point of the sample instance and the first grid center point of the first sample feature grid for the longitudinal axis offset situation;

[0362] In the prediction stage: the longitudinal axis offset data is implemented as second prediction offset data for the longitudinal axis offset situation between the predicted instance center point of the sample instance and the second grid center point of the second sample feature grid.

[0363] Channel 3: The quotient of the diagonal length of the bounding box and the diagonal length of the grid is used as the size data corresponding to the sample instance.

[0364] Wherein, in the analysis stage: the size data is implemented as the quotient of the diagonal length of the bounding box surrounding the sample instance and the diagonal length of the first grid corresponding to the first sample grid (which is the above instance size data);

[0365] In the prediction stage: the size data is realized as the quotient of the predicted diagonal length of the UI side enclosing box of the enclosing sample instance and the diagonal length of the second grid corresponding to the second sample grid (which is the above-mentioned predicted size data).

[0366] Channel 4 to Channel 39: Polar coordinate data of the instance contour analysis of the sample instance is determined by emitting 36 rays in 36 directions at preset interval angles.

[0367] In the analysis stage: polar coordinate data is implemented as sample contour polar coordinates corresponding to 36 intersection points determined by emitting 36 rays in different directions of the plane where the sample image is located with the reference position point in the sample instance as the pole. Each channel stores a sample contour polar coordinate; or, the arrangement of the channels is related to the degree of the polar angle, and based on the degree of the polar angle, each channel stores a sample contour polar diameter in a sample contour polar coordinate;

[0368] In the prediction stage: polar coordinate data is implemented as the predicted contour polar coordinates corresponding to 36 intersection points determined by emitting corresponding 36 rays in different directions of the plane where the sample image is located with the predicted reference position point in the sample instance as the pole. Each channel stores a predicted contour polar coordinate; or, the arrangement of the channels is related to the degree of the polar angle, and based on the degree of the polar angle, each channel stores a predicted contour polar diameter in the predicted contour polar coordinates.

[0369] Optionally, when the number of emitted rays changes, the number of output channels is changed by changing the number of convolution kernels. For example, when 20 rays are emitted, 36 channels for outputting polar coordinate data can be reduced to 20. This embodiment of the present application is not limited to this.

[0370] Channel 40: The result of judging whether a sample instance exists in the sample feature grid.

[0371] Wherein, in the analysis stage: the judgment result is implemented by determining whether each sample feature grid is implemented as a first sample feature grid corresponding to the sample instance;

[0372] In the prediction stage: the judgment result is implemented by predicting whether each sample feature grid is implemented as a second sample feature grid corresponding to the sample instance.

[0373] It is worth noting that the above number of channels is only an illustrative example and is not limited to this in the embodiments of the present application.

[0374] In some embodiments, the calculation of the first loss value corresponding to the first sample feature map output by the convolution layer 931 is shown in the following formula 1.

[0375] Formula 1:

[0376]

[0377] Among them, loss1 is used to represent the first loss value; α is used to represent the weight for adjusting the difference in offset data (horizontal axis offset data and vertical axis offset data) between the analysis stage and the prediction stage; β is used to represent the weight for adjusting the difference in size data between the analysis stage and the prediction stage; χ is used to represent the weight for adjusting the difference in polar coordinate data between the analysis stage and the prediction stage; δ is used to represent the weight for adjusting the difference in the first judgment result including sample instances between the analysis stage and the prediction stage; ε is used to represent the weight for adjusting the difference in the second judgment result excluding sample instances between the analysis stage and the prediction stage.

[0378] N1×N1 is used to represent the first sample feature map; i is used to represent any sample feature grid with the first grid size in the first sample feature map; 1i It is used to indicate whether the sample feature grid i includes the sample instance (determined based on the above-mentioned inclusion relationship). It can also be understood as whether the sample feature grid i is implemented as the first sample feature grid of a sample instance. If so, it is 1, otherwise it is 0; t xi Used to represent the first coordinate offset data; t xi ' is used to represent the first prediction offset data; t yi Used to represent the second coordinate offset data; t yi ' is used to represent the second prediction offset data; t si Used to represent instance size data; t si ′ is used to represent the predicted size data; j is used to represent the polar diameter in any of the 36 polar coordinates (since the correspondence between the polar angle of the sample contour and the polar angle of the predicted contour can be determined according to the same angle, the difference between the polar diameter of the sample contour and the polar diameter of the predicted contour is determined when the loss value is determined); t pij It is used to represent the jth sample contour radius corresponding to the sample instance in the sample feature grid i; t pij ′ is used to represent the jth predicted contour radius corresponding to the sample instance in the sample feature grid i; p i It is used to represent the probability of the sample instance existing in the sample feature grid i.

[0379] In some embodiments, the calculation of the second loss value corresponding to the second sample feature map output by the convolution layer 932 and the third loss value corresponding to the third sample feature map output by the convolution layer 933 refer to the calculation formula of the first loss value mentioned above, wherein the number, size and other contents of the sample feature grids vary; optionally, the weights used to adjust the differences between different data may also be appropriately adjusted, which is not limited in the embodiments of the present application.

[0380] Schematically, the first loss value corresponding to the first sample feature map output by the convolution layer 931, the second loss value corresponding to the second sample feature map output by the convolution layer 932, and the third loss value corresponding to the third sample feature map output by the convolution layer 933 are integrated to obtain a total loss value as shown in the following formula 2.

[0381] Formula 2:

[0382] loss=loss1+loss2+loss3

[0383] Among them, loss is the loss value; loss1 is the first loss value; loss2 is the second loss value; loss3 is the third loss value. This loss value combines the losses corresponding to sample feature maps of different scales.

[0384] Optionally, the loss value is iteratively optimized until the loss value of the image segmentation model no longer decreases, thereby obtaining an image segmentation model that can more accurately analyze the instance contour (shape), coordinates (position), and size.

[0385] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0386] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in instance contour recognition, avoiding the limitation of the overlap ratio determination method that cannot distinguish sample instances with similar shapes, thereby improving the accuracy and efficiency of instance segmentation.

[0387] In the embodiments of the present application, the contents of sample feature maps corresponding to different sampling multiples are obtained through an image segmentation model. With the help of sample feature maps of different feature map sizes, sample instances of different sizes can be analyzed more specifically through sample feature grids of different grid sizes; in addition, by introducing the concept of intersection-over-union ratio, a first sample feature grid that is more suitable for analyzing the sample instance can be arranged, so that when there are multiple sample instances, the corresponding sample instances can be analyzed with the help of sample feature grids corresponding to the multiple sample instances. Then, the losses corresponding to the multiple sample feature maps can be combined to obtain the loss value, thereby improving the training accuracy of the image segmentation model.

[0388] In an optional embodiment, after the image segmentation model is obtained, the image instance in the image to be identified is segmented by the image segmentation model, thereby obtaining an instance segmentation result corresponding to the image to be identified. Fig.12 As shown above Figure 2 The illustrated step 240 may then be implemented as the following steps 1210 to 1240 .

[0389] Step 1210: receiving images to be recognized sent from multiple terminals.

[0390] The image to be identified is an image used for instance identification. Optionally, the image to be identified includes at least one image instance.

[0391] Illustratively, the server receives the images to be identified sent from multiple terminals; or, another terminal device (such as a mobile phone, a computer, etc.) connected to the multiple terminals receives the images to be identified sent from the multiple terminals.

[0392] Step 1220: Determine a first image sent by a first terminal from a plurality of images to be identified.

[0393] Illustratively, the first terminal is implemented as any terminal; or, the first terminal is implemented as a main test machine, etc.

[0394] The image to be identified sent by the first terminal is used as the first image.

[0395] In some embodiments, after receiving images to be identified sent by multiple terminals, one terminal is randomly selected as the first terminal and the first image is determined; or, after receiving images to be identified sent by multiple terminals, in response to one terminal among the multiple terminals receiving a trigger operation, the terminal is used as the first terminal and the image to be identified sent by the first terminal is determined as the first image.

[0396] Step 1230, in response to the first terminal receiving a trigger operation for the first image instance, a second image other than the first image of multiple images to be identified is predicted through the trained image segmentation model to obtain an instance prediction result for instance labeling of at least one image instance.

[0397] The first image instance is an image instance in the first image on which a trigger operation is performed. Schematically, taking an image segmentation model in a computer connected to multiple terminals as an example, after the computer detects that the first terminal receives a trigger operation for the first image instance, the image segmentation model is used to predict a second image of the multiple images to be identified except the first image, thereby obtaining an instance prediction result for each instance annotation of at least one image instance in the second image.

[0398] Illustratively, at least one image instance is individually labeled by an image segmentation model, thereby distinguishing multiple predicted image instances.

[0399] The instance prediction result includes a predicted triggering area for predicting an image instance in the second image based on the first image instance.

[0400] Optionally, obtain a first instance feature corresponding to the first image instance, obtain a second instance feature corresponding to at least one image instance in the second image, compare the distance between the first instance feature and the second instance feature in the vector space, and determine the second instance feature that is closest to the first instance feature, and use the image instance corresponding to the second instance feature as the predicted trigger area corresponding to the first image instance.

[0401] Illustratively, the example features include at least one of a variety of features such as color features, shape features, and text features, which are not limited here.

[0402] Step 1240: Send a trigger instruction to a second terminal corresponding to the second image based on the instance prediction result.

[0403] The trigger instruction is used to trigger the predicted trigger area on the second terminal.

[0404] Illustratively, after determining the predicted trigger area in the instance prediction result corresponding to the second image, a trigger instruction is sent to the second terminal corresponding to the second image to realize the process of triggering the predicted trigger area on the second terminal in an indirect manner, thereby improving the trigger efficiency.

[0405] In some embodiments, if the image to be identified does not include an image instance, the instance prediction result may also be implemented as prompt information that does not include the image instance, or as an image without instance annotations, which is not limited in the embodiments of the present application.

[0406] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in identifying instance contours, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation.

[0407] In an embodiment of the present application, it is introduced that images to be identified sent by multiple terminals are identified so that a predicted trigger area corresponding to the first image instance can be determined from a second image based on a first image instance in the first image, and then the predicted trigger area on the second terminal is triggered by sending a trigger instruction, which can achieve the purpose of operating multiple second terminals by operating one first terminal, thereby improving operating efficiency.

[0408] In an optional embodiment, the above method is applied to the trigger synchronization field of multiple terminals. For example, after the game control M in the game application is triggered based on the main test machine (main control terminal), the predicted trigger area corresponding to the game control M on other terminals is determined by the image instance segmentation method, and the predicted trigger area is triggered, thereby realizing the process of jointly triggering the game applications on multiple terminals and testing the performance of the game applications based on the separate triggering of the main test machine. The above image instance segmentation method is called a multi-machine synchronization method based on polar coordinate instance segmentation. Fig.13 As shown, the above image instance segmentation method can also be implemented as the following steps 1310 to 1350.

[0409] Step 1310, construct a game control instance segmentation database.

[0410] Schematically, multiple game applications are aggregated and various interfaces of the game lobby are traversed to collect a large number of game screenshots; each game control is instance segmented by manual labeling, that is, pixel-level labeling is performed for each game control.

[0411] like Fig.14 As shown, it is a schematic diagram of a lobby interface in a game, which includes multiple game controls, such as: game control 1410, game control 1420, etc. Different game controls realize different game functions.

[0412] Similarly, if Fig.15 As shown, it is another schematic diagram of the lobby interface in the game, which also includes multiple game controls, such as: game control 1510, game control 1520, etc.

[0413] To Fig.14 For example, different pixel-level annotations are performed on different game controls in the game screenshot shown in the figure. Fig.16 As shown, it is an example after annotation. Different game controls are annotated with different pixel levels, such as: using a first pixel value to assign a value to the game control 1610 to implement the pixel-level annotation process of the game control 1610, using a second pixel value different from the first pixel value to assign a value to the game control 1620 to implement the pixel-level annotation process of the game control 1620, etc.

[0414] Different game controls are distinguished based on the pixel-level annotation process, and the annotated game screenshots are called sample images, where the divided game controls are the sample instances mentioned above. The instance segmentation process can obtain richer information, including not only coordinate position, size and other information, but also instance contour (shape) information of different game controls.

[0415] The game control instance segmentation database is used to describe a set of sample images consisting of multiple sample images that have been annotated at the pixel level.

[0416] Step 1320, training an image segmentation model based on polar coordinates.

[0417] Illustratively, the image segmentation model is trained through the above-mentioned sample image set, and the instance contour information of the game control is expressed based on pixel-level annotation. Therefore, with the help of the sample contour polar coordinates representing the instance contour and the predicted contour polar coordinates predicted by the image segmentation model, the image segmentation model can be trained more comprehensively to obtain the image segmentation model.

[0418] Optionally, after completing pixel-level annotation to obtain sample images, the image segmentation model is trained based on the sample images and the corresponding instance segmentation results (sample instances). Considering the speed and computational complexity during testing, the following is designed: Fig. 9 The lightweight model structure shown introduces a shape feature description method based on polar coordinates to train the image segmentation model.

[0419] The process of performing model training with polar coordinates has been described above. Figure 4 The corresponding embodiments and Figure 5 The corresponding embodiments have been introduced and will not be described again here.

[0420] In some embodiments, in addition to training the image segmentation model based on polar coordinates, the image segmentation model is more comprehensively trained with the help of offset data representing position and dimension data representing dimension, thereby obtaining an image segmentation model with more accurate prediction effect.

[0421] Step 1330, obtaining the multi-machine game instance segmentation result.

[0422] In an illustrative manner, multiple mobile terminals can be connected simultaneously through a computer or other device. After opening the game application through the mobile terminals, the game screens of multiple mobile terminals can be captured at the same time, and the instance segmentation results can be obtained through the trained image segmentation model (also called instance segmentation model).

[0423] These include: whether there are game controls in the sample feature grid in the sample feature map under different sampling multiples; if there are game controls, the scale, coordinate offset and normalized polar coordinates of the game controls are returned, and based on these data, the instance segmentation results of the game buttons can be obtained.

[0424] Step 1340, multi-machine game button matching.

[0425] Schematically, after the tester clicks a game control on the main test machine, the tester can obtain the clicked position through the debugging tool (Android Debug Bridge, ADB) and determine which instance segmentation area corresponds to which game control the position belongs to.

[0426] In some embodiments, after finding the instance segmentation area corresponding to the game button, the histogram features of the red, green and blue (RGB) three channels of the instance segmentation area are obtained; the text information contained in the instance segmentation area is obtained, such as: obtaining the text therein through an optical character recognition (OCR) recognition interface; obtaining the shape (instance contour) features of the game control in the instance segmentation area, etc.

[0427] Among them, the histogram features can be obtained by extracting features from the instance segmentation region (such as obtaining the histogram features through an encoder); similarly, the shape features can also be obtained by extracting features from the instance segmentation region.

[0428] Optionally, on the game image captured by other mobile phone terminals, after performing a proportional image scaling transformation on the area corresponding to the above-mentioned instance segmentation area, the instance segmentation results within 200 pixels from the transformed position are selected, and the above-mentioned features are extracted for these instance segmentation results, namely: the histogram features, text information, and shape features of the instance segmentation results on other mobile phone terminals.

[0429] Among them, if the instance segmentation results on other mobile phone terminals contain the same text as the instance segmentation area on the main test machine, the instance segmentation results are directly selected as the area corresponding to the instance segmentation area on the main test machine; otherwise, the Euclidean distance between the histogram features of the instance segmentation results on other mobile phone terminals and the histogram features of the instance segmentation area on the main test machine is calculated, and the Euclidean distance between the shape features of the instance segmentation results on other mobile phone terminals and the shape features of the instance segmentation area on the main test machine is calculated, so as to combine the two Euclidean distances and select the instance segmentation result with the closest distance as the area corresponding to the instance segmentation area on the main test machine, etc.

[0430] Step 1350, multiple machines perform actions synchronously.

[0431] Optionally, after matching corresponding segmented areas for different mobile terminals through step 1340, the center position of the area corresponding to the instance segmentation result is clicked through ADB to achieve synchronous operation of multiple machines. Through the above method, one tester can operate multiple mobile terminals at the same time, greatly improving test efficiency.

[0432] In an optional embodiment, multiple terminal devices transmit the current game screen to a computer end (such as a computer, server, etc.) in real time, and the instance segmentation results of the game image can be obtained through the image segmentation model configured in the computer end.

[0433] In order to calibrate the game buttons, the correspondence between the game controls of different terminal devices is determined based on the shape features, color features, texture features, etc. That is, the computer can determine the correspondence between different game controls in real time based on the game screen received in real time.

[0434] Optionally, when operating a game on the main test machine (i.e., the terminal device controlled by the tester), the click position is obtained through ADB. According to the instance segmentation result, the game control position corresponding to the click position can be found on the main test machine; then, according to the corresponding relationship determined in real time and the instance segmentation result, the corresponding trigger position can be found on other terminal devices other than the main test machine, and then ADB clicks the corresponding trigger area on other terminal devices, so that a single tester can control multiple terminal devices at the same time. This process will greatly improve the test efficiency.

[0435] It is worth noting that the above are only illustrative examples, and the above method can also be applied to various fields such as image content recognition, terminal performance testing, etc., and the embodiments of the present application are not limited to this.

[0436] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in identifying instance contours, avoiding the limitation of using the overlap ratio determination method to distinguish sample instances with similar shapes, and improving the accuracy and efficiency of instance segmentation.

[0437] In the embodiments of the present application, the effect achieved by applying the image instance segmentation method to the multi-machine testing scenario is introduced. Multi-machine game testing is a very important task, which can be used for multiple tasks such as game compatibility performance testing, flash back testing, and screen abnormality testing. The image segmentation model trained with the image instance segmentation method can more accurately detect the game controls in the game and perform instance segmentation; when operating multiple machines, the game controls are matched through instance features such as text information, color (histogram) features, and shape features. The accuracy of game control matching can be effectively improved, so that a tester can achieve the purpose of triggering multiple mobile terminals by operating one mobile terminal, effectively improving the testing efficiency of terminals and applications.

[0438] Fig.17 is a structural block diagram of an image instance segmentation device provided by an exemplary embodiment of the present application, such as Fig.17 As shown, the device includes the following parts:

[0439] An image acquisition module 1710 is used to acquire a sample image, wherein the sample image includes a sample instance to be identified, and the sample instance is annotated with sample contour polar coordinates for characterizing the instance contour;

[0440] A contour prediction module 1720, configured to predict the instance contour of the sample instance in the sample image by using an image segmentation model to obtain a predicted instance contour corresponding to the sample instance, wherein the image segmentation model is a model to be trained;

[0441] A coordinate prediction module 1730 is used to obtain the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour, taking the predicted reference position point in the predicted instance contour as a polar point;

[0442] A loss acquisition module 1740, configured to acquire a loss value corresponding to the sample instance based on a difference between the sample contour polar coordinates and the predicted contour polar coordinates;

[0443] The model training module 1750 is used to train the image segmentation model with the loss value to obtain a trained image segmentation model, and the image segmentation model is used to perform segmentation prediction on the image instance in the image to be identified and obtain the instance prediction result.

[0444] In an optional embodiment, the coordinate prediction module 1730 is also used to determine the predicted instance bounding box that surrounds the predicted instance contour from the sample image; based on the predicted instance bounding box diagonal of the predicted instance bounding box, determine the center point of the predicted instance bounding box as the predicted reference position point in the predicted instance contour; with the predicted reference position point as the pole, emit the rays in multiple directions of the plane where the sample image is located; based on the intersection between the multiple rays and the predicted instance contour, obtain the predicted contour polar coordinates corresponding to the multiple contour points respectively.

[0445] In an optional embodiment, the coordinate prediction module 1730 is also used to determine the polar axis in the sample image based on the pole and establish a polar coordinate system; obtain the multiple contour points based on the intersection points between the multiple rays and the predicted instance contour, and obtain the predicted contour polar coordinates of the multiple intersection points in the polar coordinate system.

[0446] In an optional embodiment, the sample contour polar coordinates are polar coordinates determined by emitting multiple rays in different directions of the plane where the sample image is located with a reference position point in the sample instance as the pole, and the reference position point corresponds to the predicted reference position point;

[0447] The coordinate prediction module 1730 is also used to determine the ray distribution law of multiple rays emitted when the polar coordinates of the sample contour are determined, and to emit the multiple rays in different directions of the plane where the sample image is located with the predicted reference position point in the predicted instance contour as the pole.

[0448] In an optional embodiment, the polar coordinates include a polar diameter and a polar angle;

[0449] The loss acquisition module 1740 is also used to determine the sample contour polar diameters and sample contour polar angles corresponding to multiple sample contour polar coordinates respectively; determine the predicted contour polar diameters and predicted contour polar angles corresponding to multiple predicted contour polar coordinates respectively, wherein there is a one-to-one correspondence between the multiple predicted contour polar angles and the multiple sample contour polar angles; and obtain the loss value corresponding to the sample instance based on the difference between the sample contour polar diameter and the predicted contour polar diameter under the corresponding relationship between the multiple predicted contour polar angles and the multiple sample contour polar angles.

[0450] In an optional embodiment, the loss acquisition module 1740 is also used to determine a first predicted contour polar angle in the first predicted contour polar coordinates among the multiple predicted contour polar coordinates; determine a first sample contour polar angle having the same angle as the first predicted contour polar angle from the sample contour polar angles corresponding to the multiple sample contour polar coordinates; determine a first predicted contour polar diameter corresponding to the first predicted contour polar angle, and determine a first sample contour polar diameter corresponding to the first sample contour polar angle; and obtain the loss value corresponding to the sample instance based on the difference between the first predicted contour polar diameter and the first sample contour polar diameter.

[0451] In an optional embodiment, the loss acquisition module 1740 is also used to obtain a contour deviation value based on the contour difference between the sample contour polar coordinates and the predicted contour polar coordinates used to describe the shape; perform feature compression on the sample image through the image segmentation model to obtain a sample feature map, and the sample feature map is composed of multiple sample feature grids; based on the instance position of the sample instance in the sample image, determine a first sample feature grid including the sample instance from the multiple sample feature grids; based on the position offset relationship between the sample instance and the first sample feature grid, determine a coordinate deviation value; based on the coordinate deviation value and the contour deviation value, obtain the loss value corresponding to the sample instance.

[0452] In an optional embodiment, the loss acquisition module 1740 is also used to determine the position offset relationship between the sample instance and the first sample feature grid based on the instance position to obtain coordinate offset data; predict the predicted instance position of the sample instance in the sample image through the image segmentation model; obtain a second sample feature grid from the multiple sample feature grids based on the predicted instance position; obtain predicted offset data based on the position offset relationship between the predicted instance position and the second sample feature grid; and obtain the coordinate deviation value through the coordinate offset data and the predicted offset data.

[0453] In an optional embodiment, the loss acquisition module 1740 is also used to determine, from the sample image, an instance bounding box surrounding the sample instance based on the instance position; determine, based on the bounding box diagonal of the instance bounding box, a center point of the instance bounding box as the instance center point of the sample instance; determine a first grid center point corresponding to the first sample feature grid; and obtain the coordinate offset data based on a position offset relationship between the instance center point and the first grid center point.

[0454] In an optional embodiment, the loss acquisition module 1740 is also used to determine instance size data corresponding to the sample instance based on the instance area of ​​the sample instance in the sample image, and the instance size data is used to characterize the instance size of the sample instance in the sample image; predict the instance size corresponding to the sample instance by the image segmentation model to obtain predicted size data; obtain a size deviation value based on the instance size data and the predicted size data; and obtain the loss value corresponding to the sample instance based on the size deviation value, the coordinate deviation value and the contour deviation value.

[0455] In an optional embodiment, the loss acquisition module 1740 is also used to determine the grid diagonal length corresponding to the first sample feature grid; determine the instance bounding box surrounding the sample instance based on the instance position; based on the bounding box diagonal length corresponding to the instance bounding box, determine the quotient of the bounding box diagonal length and the grid diagonal length to obtain the instance size data corresponding to the sample instance.

[0456] In an optional embodiment, the loss acquisition module 1740 is also used to perform feature compression on the sample image at a first sampling multiple through the image segmentation model to obtain a first sample feature map, wherein the first sample feature map includes multiple sample feature grids with a first grid size, and the first grid size is related to the first sampling multiple; perform feature compression on the sample image at a second sampling multiple through the image segmentation model to obtain a second sample feature map, wherein the second sample feature map includes multiple sample feature grids with a second grid size, the second grid size is related to the second sampling multiple, and the second sampling multiple is different from the first sampling multiple, and the second grid size is different from the first grid size.

[0457] In an optional embodiment, the loss acquisition module 1740 is also used to determine an instance bounding box surrounding the sample instance based on the instance position of the sample instance in the sample image; determine the center point of the instance bounding box as the instance center point of the sample instance based on the bounding box diagonal of the instance bounding box; determine the sample feature grid where the instance center point is located as the first instance feature grid from multiple sample feature grids having the first grid size; determine a first area intersection between the first instance feature grid and the sample instance, and determine a first area union between the first instance feature grid and the sample instance; in response to a ratio between the first area intersection and the first area union reaching a preset threshold, use the first instance feature grid as the first sample feature grid including the sample instance.

[0458] In an optional embodiment, the loss acquisition module 1740 is also used to determine an instance enclosing area surrounding the sample instance based on the instance position of the sample instance in the sample image; determine a center point of the enclosing area as the instance center point of the sample instance based on the area diagonal of the instance enclosing area; determine a sample feature grid where the instance center point is located from a plurality of sample feature grids having the second grid size as a second instance feature grid; determine a second area intersection between the second instance feature grid and the sample instance, and determine a second area union between the second instance feature grid and the sample instance; in response to a ratio between the second area intersection and the second area union reaching a preset threshold, use the second instance feature grid as the first sample feature grid including the sample content.

[0459] In an optional embodiment, the loss acquisition module 1740 is also used to determine a first sample feature grid corresponding to each sample instance from the multiple sample feature grids based on the instance positions corresponding to the multiple sample instances in the sample image; in response to the first sample feature grid being a sample feature grid with a first grid size, based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtain a first loss value corresponding to the sample instance under the first sample feature map; in response to the first sample feature grid being a sample feature grid with a second grid size, based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtain a second loss value corresponding to the sample instance under the second sample feature map; and use the sum of the first loss value and the second loss value as the loss value corresponding to the sample instance.

[0460] like Fig.18 As shown, in an optional embodiment, the device further includes:

[0461] The model recognition module 1760 is used to receive images to be recognized sent from multiple terminals respectively, wherein the images to be recognized include at least one image instance; determine a first image sent by a first terminal from the multiple images to be recognized, wherein the first image includes a first image instance; in response to the first terminal receiving a trigger operation for the first image instance, predict a second image other than the first image of the multiple images to be recognized through the trained image segmentation model to obtain the instance prediction results for instance labeling at least one image instance respectively, wherein the instance prediction results include a predicted trigger area for predicting an image instance in the second image based on the first image instance; and send a trigger instruction to a second terminal corresponding to the second image based on the instance prediction result, wherein the trigger instruction is used to trigger the predicted trigger area on the second terminal.

[0462] To summarize, on the basis of analyzing the sample instance with the help of the predicted instance contour, the predicted instance contour is expressed more carefully and intuitively by introducing polar coordinates. The sample contour polar coordinates obtained by analyzing the sample instance and the predicted contour polar coordinates obtained by predicting the sample instance can be combined to show the loss in the form of coordinate offset, and then the image segmentation model is trained with the loss value to obtain a trained image segmentation model that is more accurate in instance contour recognition, avoiding the limitation of the overlap ratio determination method that cannot distinguish sample instances with similar shapes, thereby improving the accuracy and efficiency of instance segmentation.

[0463] It should be noted that the image instance segmentation device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image instance segmentation device provided in the above embodiment and the image instance segmentation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0464] Fig.19The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown. The server 1900 includes a central processing unit (CPU) 1901, a system memory 1904 including a random access memory (RAM) 1902 and a read-only memory (ROM) 1903, and a system bus 1905 connecting the system memory 1904 and the central processing unit 1901. The server 1900 also includes a mass storage device 1906 for storing an operating system 1913, an application program 1914, and other program modules 1915.

[0465] The mass storage device 1906 is connected to the central processing unit 1901 through a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1906 and its associated computer-readable media provide non-volatile storage for the server 1900. That is, the mass storage device 1906 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0466] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. The above-mentioned system memory 1904 and mass storage device 1906 can be collectively referred to as memory.

[0467] According to various embodiments of the present application, the server 1900 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1900 can be connected to the network 1912 through the network interface unit 1911 connected to the system bus 1905, or the network interface unit 1911 can be used to connect to other types of networks or remote computer systems (not shown).

[0468] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0469] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the image instance segmentation method provided by the above-mentioned method embodiments.

[0470] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the image instance segmentation method provided by the above-mentioned method embodiments.

[0471] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image instance segmentation method described in any of the above embodiments.

[0472] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for segmenting an image instance, characterized in that: The method comprises: Acquire a sample image, wherein the sample image includes a sample instance to be identified, and the sample instance is annotated with sample contour polar coordinates for characterizing the instance contour; Predicting the instance contour of the sample instance in the sample image by using an image segmentation model to obtain a predicted instance contour corresponding to the sample instance, wherein the image segmentation model is a model to be trained; Taking the predicted reference position point in the predicted instance contour as the pole, obtaining the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour; Based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates, obtaining a loss value corresponding to the sample instance; The image segmentation model is trained with the loss value to obtain a trained image segmentation model.

2. The method according to claim 1, characterized in that: The step of taking the predicted reference position point in the predicted instance contour as a pole and obtaining the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour respectively comprises: Determining the predicted instance bounding box surrounding the predicted instance contour from the sample image; Based on a prediction bounding box diagonal of the prediction instance bounding box, determining a center point of the prediction instance bounding box as the prediction reference position point in the prediction instance contour; Taking the predicted reference position point as the pole, emitting the rays respectively in multiple directions of the plane where the sample image is located; Based on the intersection points between the multiple rays and the predicted instance contour, the predicted contour polar coordinates corresponding to the multiple contour points are obtained.

3. The method according to claim 2, characterized in that The obtaining, based on the intersection points between the multiple rays and the predicted instance contour, the predicted contour polar coordinates corresponding to the multiple contour points respectively includes: Determine the polar axis in the sample image based on the pole and establish a polar coordinate system; The plurality of contour points are obtained based on the intersection points between the plurality of rays and the predicted instance contour, and the polar coordinates of the predicted contour of the plurality of intersection points in the polar coordinate system are obtained.

4. The method according to claim 2, characterized in that: The sample contour polar coordinates are polar coordinates determined by emitting a plurality of rays in different directions of the plane where the sample image is located with a reference position point in the sample instance as a pole, and the reference position point corresponds to the predicted reference position point; The step of taking the predicted reference position point in the predicted instance contour as a pole and obtaining the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour respectively comprises: By determining the ray distribution law of multiple rays emitted when the polar coordinates of the sample contour are determined, the multiple rays are emitted in different directions of the corresponding plane of the sample image with the predicted reference position point in the predicted instance contour as the pole.

5. The method according to any one of claims 1 to 4, characterized in that: The acquiring the loss value corresponding to the sample instance based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates includes: Determine the sample contour polar diameters and sample contour polar angles corresponding to the plurality of sample contour polar coordinates; Determine the predicted contour polar diameters and predicted contour polar angles corresponding to the plurality of predicted contour polar coordinates, wherein the plurality of predicted contour polar angles correspond to the plurality of sample contour polar angles in one-to-one correspondence; Under the corresponding relationship between the multiple predicted contour polar angles and the multiple sample contour polar angles, the loss value corresponding to the sample instance is obtained based on the difference between the sample contour polar diameter and the predicted contour polar diameter.

6. The method according to claim 5, characterized in that The step of acquiring the loss value corresponding to the sample instance based on the difference between the sample contour polar diameter and the predicted contour polar diameter under the corresponding relationship between the multiple predicted contour polar angles and the multiple sample contour polar angles includes: For a first predicted contour polar coordinate among the plurality of predicted contour polar coordinates, determining a first predicted contour polar angle in the first predicted contour polar coordinate; Determine, from the sample contour polar angles respectively corresponding to the plurality of sample contour polar coordinates, a first sample contour polar angle having the same angle as the first predicted contour polar angle; Determine a first predicted contour polar diameter corresponding to the first predicted contour polar angle, and determine a first sample contour polar diameter corresponding to the first sample contour polar angle; Based on the difference between the first predicted contour pole diameter and the first sample contour pole diameter, the loss value corresponding to the sample instance is obtained.

7. The method according to any one of claims 1 to 4, characterized in that: The acquiring the loss value corresponding to the sample instance based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates comprises: Obtaining a contour deviation value based on a contour difference between the sample contour polar coordinates and the predicted contour polar coordinates for describing the shape; Performing feature compression on the sample image through the image segmentation model to obtain a sample feature map, wherein the sample feature map is composed of a plurality of sample feature grids; Based on the instance position of the sample instance in the sample image, determining a first sample feature grid including the sample instance from the plurality of sample feature grids; Determining a coordinate deviation value based on a position offset relationship between the sample instance and the first sample feature grid; Based on the coordinate deviation value and the contour deviation value, the loss value corresponding to the sample instance is obtained.

8. The method according to claim 7, characterized in that The determining of the coordinate deviation value based on the position offset relationship between the sample instance and the first sample feature grid includes: Determine the position offset relationship between the sample instance and the first sample feature grid based on the instance position to obtain coordinate offset data; Predicting a predicted instance position of the sample instance in the sample image by using the image segmentation model; Acquire a second sample feature grid from the plurality of sample feature grids based on the predicted instance position; Obtaining prediction offset data based on a position offset relationship between the predicted instance position and the second sample feature grid; The coordinate deviation value is obtained through the coordinate offset data and the predicted offset data.

9. The method according to claim 8, characterized in that The determining, based on the instance position, a position offset relationship between the sample instance and the first sample feature grid to obtain coordinate offset data comprises: Based on the instance position, determining an instance bounding box surrounding the sample instance from the sample image; Based on the diagonal of the instance bounding box, determining the center point of the instance bounding box as the instance center point of the sample instance; Determine a first grid center point corresponding to the first sample feature grid; The coordinate offset data is obtained based on the position offset relationship between the instance center point and the first grid center point.

10. The method according to claim 7, characterized in that The acquiring the loss value corresponding to the sample instance based on the coordinate deviation value and the contour deviation value comprises: Determine instance size data corresponding to the sample instance based on an instance region of the sample instance in the sample image, wherein the instance size data is used to characterize an instance size of the sample instance in the sample image; Predicting the instance size corresponding to the sample instance by the image segmentation model to obtain predicted size data; Based on the instance size data and the predicted size data, obtaining a size deviation value; Based on the size deviation value, the coordinate deviation value, and the contour deviation value, the loss value corresponding to the sample instance is acquired.

11. The method according to claim 10, characterized in that The determining, based on the instance region of the sample instance in the sample image, instance size data corresponding to the sample instance includes: Determine the diagonal length of the grid corresponding to the first sample feature grid; Determine an instance bounding box surrounding the sample instance based on the instance position; Based on the diagonal length of the bounding box corresponding to the instance bounding box, the quotient of the diagonal length of the bounding box and the diagonal length of the grid is determined to obtain the instance size data corresponding to the sample instance.

12. The method according to claim 7, characterized in that The step of compressing the features of the sample image by using the image segmentation model to obtain a sample feature map includes: Performing feature compression on the sample image at a first sampling multiple through the image segmentation model to obtain a first sample feature map, wherein the first sample feature map includes a plurality of sample feature grids having a first grid size, and the first grid size is related to the first sampling multiple; The sample image is feature compressed by the image segmentation model at a second sampling multiple to obtain a second sample feature map, wherein the second sample feature map includes a plurality of sample feature grids having a second grid size, wherein the second grid size is related to the second sampling multiple, and the second sampling multiple is different from the first sampling multiple, and the second grid size is different from the first grid size.

13. The method according to claim 12, characterized in that The determining, based on the instance position of the sample instance in the sample image, a first sample feature grid including the sample instance from the multiple sample feature grids comprises: Determine an instance bounding box surrounding the sample instance based on the instance position of the sample instance in the sample image; Based on the diagonal of the instance bounding box, determining the center point of the instance bounding box as the instance center point of the sample instance; Determine, from a plurality of sample feature grids having the first grid size, a sample feature grid where the instance center point is located as a first instance feature grid; Determine a first area intersection between the first instance feature grid and the sample instance, and determine a first area union between the first instance feature grid and the sample instance; In response to a ratio between the first region intersection and the first region union reaching a preset threshold, the first instance feature grid is used as the first sample feature grid including the sample instance.

14. The method according to claim 12, characterized in that The determining, based on the instance position of the sample instance in the sample image, a first sample feature grid including the sample instance from the multiple sample feature grids comprises: Determine an instance enclosing region enclosing the sample instance based on the instance position of the sample instance in the sample image; Based on the area diagonal of the instance enclosing area, determining the center point of the enclosing area as the instance center point of the sample instance; Determine, from a plurality of sample feature grids having the second grid size, a sample feature grid where the instance center point is located as a second instance feature grid; Determine a second area intersection between the second instance feature grid and the sample instance, and determine a second area union between the second instance feature grid and the sample instance; In response to a ratio between the second area intersection and the second area union reaching a preset threshold, the second instance feature grid is used as the first sample feature grid including the sample content.

15. The method according to claim 12, characterized in that The determining, based on the instance position of the sample instance in the sample image, a first sample feature grid including the sample instance from the multiple sample feature grids comprises: Based on instance positions respectively corresponding to the plurality of sample instances in the sample image, determining a first sample feature grid respectively corresponding to each sample instance from the plurality of sample feature grids; The acquiring the loss value corresponding to the sample instance based on the difference between the sample contour polar coordinates and the predicted contour polar coordinates comprises: In response to the first sample feature grid being a sample feature grid having a first grid size, obtaining a first loss value corresponding to the sample instance under the first sample feature map based on a difference between the sample contour polar coordinates and the predicted contour polar coordinates; In response to the first sample feature grid being a sample feature grid having a second grid size, obtaining a second loss value corresponding to the sample instance under the second sample feature map based on a difference between the sample contour polar coordinates and the predicted contour polar coordinates; The sum of the first loss value and the second loss value is used as the loss value corresponding to the sample instance.

16. The method according to any one of claims 1 to 3, characterized in that: After the image segmentation model is trained with the loss value to obtain the trained image segmentation model, the method further includes: Receiving images to be identified sent from a plurality of terminals respectively, wherein the images to be identified include at least one image instance; Determine a first image sent by a first terminal from a plurality of images to be identified, wherein the first image includes a first image instance; In response to the first terminal receiving a trigger operation for the first image instance, predicting a second image other than the first image among the multiple images to be identified by using the trained image segmentation model to obtain the instance prediction result for respectively labeling at least one image instance, wherein the instance prediction result includes a predicted trigger area for predicting an image instance in the second image based on the first image instance; A trigger instruction is sent to a second terminal corresponding to the second image based on the instance prediction result, where the trigger instruction is used to trigger the predicted trigger area on the second terminal.

17. A device for segmenting an image instance, characterized in that: The device comprises: An image acquisition module, used for acquiring a sample image, wherein the sample image includes a sample instance to be identified, and the sample instance is annotated with sample contour polar coordinates for characterizing the instance contour; A contour prediction module, used for predicting the instance contour of the sample instance in the sample image by using an image segmentation model to obtain a predicted instance contour corresponding to the sample instance, wherein the image segmentation model is a model to be trained; A coordinate prediction module, used to obtain the predicted contour polar coordinates corresponding to a plurality of contour points on the predicted instance contour, taking the predicted reference position point in the predicted instance contour as a pole; A loss acquisition module, configured to acquire a loss value corresponding to the sample instance based on a difference between the sample contour polar coordinates and the predicted contour polar coordinates; The model training module is used to train the image segmentation model with the loss value to obtain a trained image segmentation model.

18. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the image instance segmentation method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the image instance segmentation method as described in any one of claims 1 to 16.

20. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the image instance segmentation method according to any one of claims 1 to 16.

Images