Image processing method, apparatus, device, readable storage medium and program product

Abstract

Embodiments of the present application provide an image processing method, device, equipment, readable storage medium and program product, which can be applied to the fields or scenarios of cloud technology, artificial intelligence, image recognition, vehicle-mounted, etc. The method comprises: performing feature extraction processing on a to-be-processed image to determine a target global feature map of the to-be-processed image; determining a target candidate region feature map corresponding to a foreground object in the to-be-processed image according to the target global feature map; inputting the target candidate region feature map into a mask branch network in an instance segmentation model for processing to determine a foreground mask corresponding to the foreground object; inputting the target candidate region feature map into a target detection branch network in the instance segmentation model for processing to determine a foreground category and a foreground position bounding box corresponding to the foreground object; and determining an instance segmentation result of the to-be-processed image according to the foreground mask, the foreground category and the foreground position bounding box corresponding to the foreground object. The embodiments of the present application can improve the accuracy of the instance segmentation result.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to image processing methods, image processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] Image instance segmentation is a technique that further refines object detection by separating the foreground and background of an object to achieve pixel-level object separation. However, existing instance segmentation networks have significant limitations in application scenarios. For example, for specific types with high sampling difficulty, they can only be trained on small datasets (i.e., small data volumes), resulting in low accuracy of the instance segmentation results obtained by the trained network. Therefore, improving the accuracy of instance segmentation results is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, readable storage medium, and program product that can improve the accuracy of instance segmentation results.

[0004] In a first aspect, this application provides an image processing method, the method comprising: Feature extraction is performed on the image to be processed to determine the target global feature map of the image to be processed. Based on the above target global feature map, a target candidate region feature map is determined. The image region corresponding to the above target candidate region feature map includes the foreground object in the above image to be processed. The above target candidate region feature map is input into the mask branch network in the instance segmentation model for processing to determine the foreground mask corresponding to the above foreground object. The above target candidate region feature map is input into the target detection branch network in the above instance segmentation model for processing to determine the foreground category and foreground location bounding box corresponding to the above foreground object; Based on the foreground mask, foreground category, and foreground location bounding box corresponding to the foreground object, the instance segmentation result of the image to be processed is determined. The instance segmentation model is trained using sample data, which includes sample images, synthesized images, a reference foreground mask for the sample images, a reference foreground category for the synthesized images, and reference foreground location bounding boxes. The synthesized images are obtained by combining foreground images extracted from the sample images with a portion of the background image. The foreground images include foreground objects from the sample images, and the foreground category of the foreground objects in the sample images is the target foreground category. The mask branch network is trained using an image segmentation loss, which is determined based on the predicted foreground mask and the reference foreground mask. The predicted foreground mask is obtained by processing the candidate region feature maps of the sample images using the mask branch network. The object detection branch network is trained using object classification loss and bounding box regression loss, whereby the object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding boxes and the reference foreground location bounding boxes. The predicted foreground category and the predicted foreground location bounding boxes are obtained by processing the candidate region feature maps of the synthesized images using the object detection branch network.

[0005] Secondly, this application provides an image processing apparatus, the apparatus comprising: The preprocessing module is used to perform feature extraction on the image to be processed and determine the target global feature map of the image to be processed. The aforementioned preprocessing module is further configured to determine a target candidate region feature map based on the aforementioned target global feature map, wherein the image region corresponding to the aforementioned target candidate region feature map includes the foreground object in the aforementioned image to be processed; The instance segmentation module is used to input the feature map of the above target candidate region into the mask branch network in the instance segmentation model for processing, and to determine the foreground mask corresponding to the above foreground object. The instance segmentation module described above is also used to input the feature map of the target candidate region into the target detection branch network in the instance segmentation model for processing, and to determine the foreground category and foreground location bounding box corresponding to the foreground object. The output module is used to determine the instance segmentation result of the image to be processed based on the foreground mask, foreground category and foreground position bounding box corresponding to the foreground object. The instance segmentation model is trained using sample data, which includes sample images, synthesized images, a reference foreground mask for the sample images, a reference foreground category for the synthesized images, and reference foreground location bounding boxes. The synthesized images are obtained by combining foreground images extracted from the sample images with a portion of the background image. The foreground images include foreground objects from the sample images, and the foreground category of the foreground objects in the sample images is the target foreground category. The mask branch network is trained using an image segmentation loss, which is determined based on the predicted foreground mask and the reference foreground mask. The predicted foreground mask is obtained by processing the candidate region feature maps of the sample images using the mask branch network. The object detection branch network is trained using object classification loss and bounding box regression loss, whereby the object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding boxes and the reference foreground location bounding boxes. The predicted foreground category and the predicted foreground location bounding boxes are obtained by processing the candidate region feature maps of the synthesized images using the object detection branch network.

[0006] Thirdly, this application provides a computer device, including: a processor, a storage device, and a communication interface, wherein the processor, the communication interface, and the storage device are interconnected, wherein the storage device stores executable program code, and the processor is used to call the executable program code to implement the image processing method described above.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that are executed by a processor to implement the image processing method described above.

[0008] Fifthly, this application provides a computer program product, which includes a computer program or computer instructions, which are executed by a processor to implement the image processing method described above.

[0009] This application first extracts the global feature map of the image to be processed, and then extracts the candidate region feature map corresponding to the foreground object in the image to be processed based on the global feature map. The image feature extraction is completed through the above two stages, improving the accuracy of feature extraction. The foreground mask of the foreground object is determined by the mask branch network, and the foreground category and foreground location bounding box of the foreground object are determined by the object detection branch network. This makes the instance segmentation result obtained by the instance segmentation model contain multi-dimensional feature information, improves the data richness of the recognition result, and thus achieves more refined recognition.

[0010] During the training phase of the instance segmentation model, the model is trained using sample images and synthetic images. Since sample images are a target type that is difficult to obtain, synthetic images are synthesized by extracting foreground images and parts of the background image from the sample images, increasing the sample dataset and improving data reuse. Furthermore, the synthetic image and the sample image include the same foreground object, ensuring feature consistency for the foreground object between the sample image and the synthetic image. Using these sample images and synthetic images as image pairs for training the instance segmentation model can improve the model training effect. Considering the significant neighborhood difference between the sample images and the synthetic images, this application only calculates image segmentation loss for the sample images and target classification loss and bounding box regression loss for the synthetic images. These losses are used to adjust and optimize the network parameters of the instance segmentation model, avoiding the impact of neighborhood bias on model training, thereby improving the recognition accuracy of the trained instance segmentation model and ultimately improving the accuracy of the instance segmentation results. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the architecture of an image processing system provided in an exemplary embodiment of this application; Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of a small sample instance segmentation training process provided in an exemplary embodiment of this application; Figure 4 This is a schematic flowchart of another image processing method provided in an exemplary embodiment of this application; Figure 5A This is a flowchart illustrating the generation of a synthetic image, provided in an exemplary embodiment of this application. Figure 5B This is a flowchart illustrating the training process of an instance segmentation model based on image pairs, provided in an exemplary embodiment of this application. Figure 6 This is a schematic block diagram of an image processing apparatus provided in an exemplary embodiment of this application; Figure 7 This is a schematic block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0015] The embodiments of the present invention can be applied to various fields or scenarios such as cloud computing, cloud IoT, cloud gaming, artificial intelligence, image recognition, vehicle scenarios, intelligent transportation, assisted driving, and industrial quality inspection. Several typical fields or scenarios will be introduced below.

[0016] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire internet model and enterprise management model. This application can store data such as instance partitioning results on a cloud server. When different data are needed, they can be directly retrieved from the cloud server, greatly improving data retrieval speed.

[0017] Intelligent Traffic Systems (ITS), also known as Intelligent Transportation Systems, effectively integrate advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing. This strengthens the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, enhances the environment, and saves energy.

[0018] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), or simply vehicle-road cooperative systems, represent a development direction for Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement comprehensive, real-time dynamic information interaction between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active safety control of vehicles and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system. This application can be applied to the aforementioned fields. For example, in autonomous driving scenarios, by acquiring real-time road condition images, the instance segmentation method of this application efficiently and accurately identifies various instance objects in the road condition images, such as vehicles, pedestrians, and obstacles. Based on the identification results, corresponding hazard avoidance operations are implemented, improving the safety and stability of autonomous driving and ensuring traffic safety.

[0019] Artificial Intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision and machine learning / deep learning. The solutions provided in this application's embodiments involve machine learning and other technologies under the umbrella of AI technology, which will be described below.

[0020] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. This application primarily relates to inductive learning techniques within machine learning. Inductive learning aims to extract general decision rules and patterns from large amounts of empirical data; it is a learning method that derives general rules from specific cases. Specifically, the method proposed in this application trains the mask branch network and object detection branch network in an initial instance segmentation model using sample data to obtain an instance segmentation model capable of performing image instance segmentation tasks. Once any image to be processed is acquired, it can be processed according to the instance segmentation model to obtain instance segmentation results such as foreground mask, foreground category, and foreground location bounding box in the image to be processed.

[0021] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition. Specifically, the method proposed in this application integrates image processing, image recognition, and image semantic understanding technologies, enabling the instance segmentation model to efficiently and accurately handle various types of instance segmentation tasks.

[0022] With the research and advancement of artificial intelligence (AI) technology, it is being studied and applied in numerous fields, such as smart homes, wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, connected vehicles, and 3D games. As technology continues to develop, AI will be applied in even more areas and play an increasingly important role.

[0023] Many instance segmentation models in open environments require instance-level detection and segmentation of novel target classes (e.g., target types not encountered during pre-training). However, existing detection and instance segmentation networks are not adept at this task, especially with a small number of labeled instances. The key issue is that instance segmentation models generally assume that there are no unlabeled potential novel target classes in small sample (limited training data) labeled images (i.e., all target types in small sample labeled images are types already added during training). This is not the case. Once unlabeled novel target classes appear in the labeled data, existing few-shot learning models teach instance segmentation models to treat these unlabeled targets as background, resulting in low accuracy.

[0024] To address this issue, this application proposes a simple yet powerful pairwise data augmentation and training scheme. Specifically, to avoid suppressing potential new classes (visible but unlabeled background targets), this application obtains labeled foreground objects from sample images and a portion of background images randomly sampled from the sample images (i.e., the original images). By pasting the foreground objects onto a portion of the background images, a completely new, simple background image (i.e., the synthesized image) is created, making the sample image and the synthesized image an image pair. The instance segmentation model is trained based on these images. However, since simple training on such synthesized images is susceptible to domain offsets—for example, the resolution and size of the synthesized image differ significantly from those of the sample image—the resulting instance segmentation model exhibits low recognition accuracy.

[0025] Therefore, this application proposes a few-sample instance segmentation training method for paired images, decoupling the traditional training into two parts. The first part trains the candidate region classification and regression heads of the object detection branch network on synthetic images; the second part trains the mask segmentation head of the mask branch network on sample images. In this way, the instance segmentation model will no longer identify potential new types of objects as background, improving recognition accuracy. At the same time, it can be well generalized to instance segmentation tasks of real images, ensuring the accuracy of instance segmentation in real business scenarios. The small-sample instance segmentation training method proposed in this application can be seamlessly applied to scenarios where sample data collection is difficult. On automated industrial production lines, current production technologies can ensure high product yields, but there is still a small possibility of defective products. To ensure all products are qualified, production workshops must use the primitive method of manual inspection (inspectors checking each product individually) to perform full-volume checks. Manual quality inspection is not only slow but also prone to fatigue. It is possible to replace human labor with intelligent quality inspection using computer vision technology. This application first proposes that intelligent quality inspection can be performed using visual recognition algorithms, such as visual deep learning technology based on convolutional neural networks. However, most of the above technologies rely on large-scale labeled data, while defective products only account for a small portion of the total, making it very difficult to collect a large number of defective samples in a short time on industrial production lines.

[0026] Based on this, this application proposes a small-sample instance segmentation defect inspection technique to solve the aforementioned problems. Since defect identification depends not only on its physical characteristics but also on its severity, more refined identification of the inspection object is required. This includes identifying not only the type of defect but also its location and size. Instance segmentation technology can effectively accomplish these tasks. The small-sample instance segmentation technique proposed in this application, based on a large amount of publicly available pre-trained data, only requires sampling a few instances of defect samples to achieve excellent identification results, greatly reducing the reliance on large-scale labeled defect sample data. In scenarios where defect samples are difficult to collect or label, the proposed method can effectively improve the accuracy of defect inspection.

[0027] It is understood that in the specific embodiments of this application, data such as sample images are involved. When the above embodiments of this application are applied to specific products or technologies, the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0028] This application will be specifically illustrated through the following embodiments: Please see Figure 1 This figure is a schematic diagram of the architecture of an image processing system provided in an exemplary embodiment of this application. The image processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network (LAN), a wide area network (WAN), or the mobile internet. The user operates on a browser or client application on the terminal device 101 to process the image to be processed. The server 102 can respond to this operation by providing various image processing-related services to the user.

[0029] In one embodiment, when the terminal device 101 needs to obtain the instance segmentation result of the image to be processed, it can send the image to be processed to the server 102. After obtaining the image to be processed, the server 102 performs feature extraction processing on the image to be processed to determine the target global feature map of the image to be processed. The server 102 determines the target candidate region feature map corresponding to the foreground object in the image to be processed based on the target global feature map. The server 102 then inputs the target candidate region feature map into the mask branch network in the instance segmentation model for processing to determine the foreground mask corresponding to the foreground object. In addition, the server 102 inputs the target candidate region feature map into the target detection branch network in the instance segmentation model for processing to determine the foreground category and foreground location bounding box corresponding to the foreground object. The server 102 generates the instance segmentation result of the image to be processed based on the foreground mask, foreground category and foreground location bounding box corresponding to the foreground object, and sends the instance segmentation result to the terminal device 101.

[0030] In one embodiment, terminal device 101 can send sample images and synthesized images used to train an instance segmentation model to server 102; server 102 performs feature extraction processing on the sample images to determine a first global feature map of the sample images, and determines a first candidate region feature map corresponding to the foreground object in the sample images based on the first global feature map; server 102 performs feature extraction processing on the synthesized images to determine a second global feature map of the synthesized images, and determines a second candidate region feature map corresponding to the foreground object in the synthesized images based on the second global feature map; server 102 inputs the first candidate region feature map into a mask branch network for processing to determine the predicted foreground mask of the foreground object in the sample images. The image segmentation loss is determined based on the predicted foreground mask and the reference foreground mask, and the network parameters of the mask branch network are adjusted according to the image segmentation loss to obtain the trained mask branch network. The server 102 inputs the feature map of the second candidate region into the object detection branch network for processing, determining the predicted foreground category and predicted foreground location bounding box of the foreground object in the synthesized image. The object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding box and the reference foreground location bounding box. The server 102 adjusts the network parameters of the object detection branch network according to the object classification loss and the bounding box regression loss to obtain the trained object detection branch network. Based on this, the server 102 can construct an instance segmentation model using the trained mask branch network and the trained object detection branch network.

[0031] Terminal equipment 101 is also referred to as terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal equipment can be smart home appliances, handheld devices with wireless communication capabilities (such as smartphones and tablets), computing devices (such as personal computers (PCs), in-vehicle terminals, smart voice interaction devices, wearable devices, or other smart devices, but is not limited to these.

[0032] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0033] In one embodiment, the architecture of the image processing system proposed in this application may further include a database. The database stores images to be processed, sample data, and data related to the instance segmentation model. This data can be recorded in different database tables. For example, the database can be a database located on a server, i.e., a database built into or integrated with the server; the database can also be an external database connected to the server, such as a cloud database (i.e., a database deployed in the cloud). Specifically, it can be deployed based on any of the following: private cloud, public cloud, hybrid cloud, edge cloud, etc., thus allowing the cloud database to focus on different functions. For example, a database deployed in a private cloud uses basic cloud hardware that is the user's personal device, focusing more on serving a small group of users. A database deployed in a public cloud, on the other hand, is deployed based on a third-party cloud platform, allowing the data stored in the database to be shared. Any user's data can be stored in this database, and any user can use the data in the database.

[0034] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, the image processing method provided in the embodiments of this application can be executed not only by server 102, but also by other servers or server clusters that are different from server 102 and can communicate with terminal device 101 and / or server 102. Those skilled in the art will understand that... Figure 1The number of terminal devices and servers shown is merely illustrative. Any number of terminal devices and servers can be configured according to business needs. Furthermore, as system architecture evolves and new business scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems. In subsequent embodiments, "terminal device" will refer to the aforementioned terminal device 101, and "server" will refer to the aforementioned server 102; further details will not be repeated in subsequent embodiments.

[0035] Please see Figure 2 This figure is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application, applied to a server (referring to...). Figure 1 Taking server 102 as an example, the method may include the following steps: S201. Perform feature extraction processing on the image to be processed to determine the target global feature map of the image to be processed.

[0036] In this embodiment, the server can perform feature extraction processing on the image to be processed to obtain a target global feature map that represents the overall image features of the image to be processed. This allows the server to determine the target candidate region feature map corresponding to the foreground object in the image to be processed based on the target global feature map, thereby realizing the image feature extraction steps in the two stages mentioned above and improving the accuracy of feature extraction.

[0037] In one embodiment, the above-mentioned step of performing feature extraction processing on the image to be processed can be performed by the server through a feature extraction network, which can be a sub-network of the instance segmentation model proposed in this application.

[0038] S202. Determine the target candidate region feature map based on the target global feature map. The image region corresponding to the target candidate region feature map includes the foreground object in the image to be processed.

[0039] In this embodiment, the target candidate region feature map can be understood as a part of the target global feature map. The target global feature map is the overall image feature extracted from the image to be processed, while the target candidate region feature map is a local image feature extracted based on the target global feature map, focusing on the image region containing the foreground object in the image to be processed. For example, the image feature of the image region formed by the bounding rectangle corresponding to the foreground object in the image to be processed. Through the above two-stage feature extraction operation, the server can accurately obtain the target candidate region feature map corresponding to the foreground object in the image to be processed, and then perform subsequent instance segmentation processing based on the target candidate region feature map, thereby improving the accuracy of the instance segmentation result.

[0040] In one embodiment, the instance segmentation model in this application may further include a candidate region generation network, which is used to extract features from the region of interest (i.e., the image region corresponding to the foreground object) in the target global feature map to obtain a target candidate region feature map. Therefore, the above-mentioned determination of the target candidate region feature map based on the target global feature map can be achieved according to the following steps: inputting the target global feature map into the candidate region generation network for processing to determine the target candidate region feature map.

[0041] In this embodiment, the candidate region generation network is trained based on the candidate region regression loss and candidate region classification loss obtained from the synthesized image. The candidate region generation network can generate candidate region bounding boxes corresponding to the foreground object, while the annotation information of the synthesized image includes the reference foreground position bounding box corresponding to the foreground object. By calculating the loss between the candidate region bounding boxes and the reference foreground position bounding boxes of the synthesized image, the candidate region regression loss can be determined.

[0042] Similarly, the candidate region generation network can generate candidate region categories for candidate region bounding boxes corresponding to foreground objects, while the annotation information of the synthesized image includes reference foreground categories for reference foreground location bounding boxes corresponding to foreground objects. By calculating the loss between the candidate region categories and the reference foreground categories of the synthesized image, the candidate region classification loss can be determined. Both the candidate region bounding boxes and candidate region categories mentioned above are obtained by processing the global feature map of the synthesized image through the candidate region generation network.

[0043] The structure and training process of the candidate region generation network will be described in detail in subsequent embodiments, and will not be repeated in this embodiment.

[0044] S203. Input the feature map of the target candidate region into the mask branch network in the instance segmentation model for processing to determine the foreground mask corresponding to the foreground object.

[0045] In this embodiment, the server performs recognition processing on the feature map of the target candidate region by calling a mask branch network, thereby obtaining the foreground mask corresponding to the foreground object. The mask branch network is used to predict the foreground mask corresponding to the target object in the feature map of the target candidate region of the image to be processed. The foreground mask can indicate information such as the contour, shape, relative position, and relative size of the foreground object in the image to be processed.

[0046] S204. Input the target candidate region feature map into the target detection branch network in the instance segmentation model for processing to determine the foreground category and foreground location bounding box corresponding to the foreground object.

[0047] In this embodiment, the server uses an object detection branch network to identify and process the feature map of the target candidate region, thereby obtaining the foreground category and foreground location bounding box corresponding to the foreground object. The object detection branch network is used to predict the foreground category and foreground location bounding box corresponding to the target object in the feature map of the target candidate region of the image to be processed. The foreground category can indicate the type of foreground object in the image to be processed (e.g., plants); the foreground location bounding box can indicate information such as the relative position and relative size of the foreground object in the image to be processed (e.g., the bounding rectangle corresponding to the foreground object in the image to be processed).

[0048] S205. Determine the instance segmentation result of the image to be processed based on the foreground mask, foreground category, and foreground position bounding box corresponding to the foreground object.

[0049] In this embodiment, by integrating the foreground mask, foreground category, and foreground position bounding box corresponding to the foreground object into the instance segmentation result, the instance segmentation result contains feature information of multiple dimensions, improving the data richness of the recognition result and thus achieving more refined recognition.

[0050] For example, the image to be processed is an image with a goat in the foreground and a lawn in the background. By performing instance segmentation on the image to be processed using an instance segmentation model, the instance segmentation result of the image to be processed can be obtained. The instance segmentation result includes the foreground mask (i.e., the outline and shape information of the goat in the image to be processed) corresponding to the foreground object (i.e., the goat), the foreground category (i.e., the foreground category is "goat"), and the foreground position marker box (i.e., the bounding rectangle corresponding to the goat in the image to be processed).

[0051] In one embodiment, this application proposes a few-shot instance segmentation training method. Based on this method, the instance segmentation model can be trained in two stages: preliminary training and few-shot new class training. The instance segmentation model trained using sample data is the instance segmentation model after preliminary training. The training process includes, but is not limited to, the following steps: In the first stage, the instance segmentation model is initially trained using base class data to obtain the pre-trained instance segmentation model. In the second stage, the pre-trained instance segmentation model is trained using sample data with few-sample new classes to obtain the instance segmentation model trained with few-sample new classes.

[0052] The base class data includes multiple base class images, where the foreground category of the foreground object differs from that of the foreground object in the sample images. Base class data refers to common and easily accessible foreground categories, such as cows, horses, cats, sunflowers, and goats. The foreground category of the foreground object in the sample images is the target foreground category, which is the foreground category corresponding to the new category data. The target foreground category refers to uncommon and difficult-to-obtain foreground categories, such as cracks, scratches, and burrs (referring to burrs generated at the joints and disjoint points of industrial parts during production). Obtaining the aforementioned new category data is more difficult than obtaining the base class data, making it less likely to acquire a large-scale, highly diverse training set of new categories. Therefore, this application, based on the difference in foreground categories between the base class data and the sample data, and using a few-shot learning method, performs a first-stage pre-training on a large amount of easily accessible base class data, followed by a second-stage model fine-tuning training on a small amount of difficult-to-obtain sample data (new category data). This allows the instance segmentation model obtained through two-stage training to possess transfer learning capabilities, improving the recognition ability of the instance segmentation model and ensuring recognition accuracy. In subsequent embodiments, this application will focus on the second stage of learning.

[0053] Please see Figure 3 This figure is a schematic diagram of a few-sample instance segmentation training process provided in an exemplary embodiment of this application. The figure includes a first stage and a second stage. In the first stage, the instance segmentation model is trained based on large-scale labeled base class data (e.g., base class data with foreground types such as cars, people, and cats). In the second stage, the instance segmentation model trained in the first stage is trained based on few-sample labeled new class data (e.g., new class data with foreground types such as cracks, scratches, and burrs). Finally, a fully trained instance segmentation model is obtained, at which point the recognition capability of the instance segmentation model is greatly enhanced.

[0054] This application employs a transfer learning paradigm for few-sample instance segmentation training. The transfer learning approach mainly consists of two steps. First, the instance segmentation model is pre-trained on a large amount of available base class data, following a method similar to the fully supervised instance segmentation training process. Second, the parameters of the instance segmentation model's backbone network are frozen, and fine-tuning is performed on a small amount of new class data. Before performing the second step, because the number of foreground categories of the foreground objects in the base class data trained in the first step (e.g., 1000 base class categories) differs from the number of foreground categories of the foreground objects in the new class data trained in the second step (e.g., 50 new class categories), the input parameters in the object detection branch network and the masking branch network of the instance segmentation model need to be modified to match the number of new class categories before proceeding with the second step. This training process can be summarized in the following two steps:

[0055] in, It refers to the base class data set. It refers to a new type of data set. , , In This represents a function mapping implemented using a deep neural network. The subscript indicates how the parameters involved in the mapping function are obtained. For instance segmentation tasks, the parameters mainly consist of two parts: backbone parameters and instance segmentation header parameters. The parameters of the backbone network are obtained after training on a large-scale base class image classification dataset (such as the ImageNet dataset), while the parameters of the instance segmentation head are obtained through random initialization. Is Based on, use Base class data set pairs All parameters were obtained through training and optimization. Is Based on this, modify the parameters related to category prediction in the instance segmentation header (e.g., the parameters corresponding to the number of predicted categories), and then use... The new dataset was obtained by further training and optimization of the adjusted parameters.

[0056] Based on the above embodiments, the beneficial effects of this application are as follows: This application first extracts the global feature map of the image to be processed, and then extracts the candidate region feature map corresponding to the foreground object in the image to be processed based on the global feature map. Image feature extraction is completed through the above two stages, improving the accuracy of feature extraction. The foreground mask of the foreground object is determined by the mask branch network, and the foreground category and foreground position bounding box of the foreground object are determined by the object detection branch network. This makes the instance segmentation result obtained by the instance segmentation model contain multi-dimensional feature information, improving the data richness of the recognition result, thereby achieving more refined recognition.

[0057] In the training phase of the instance segmentation model, the model is trained using sample images and synthetic images. Since the sample images represent target types that are difficult to collect or acquire, synthetic images are obtained by extracting foreground images and part of the background image from the sample images, increasing the sample dataset and improving data reuse. Furthermore, the synthetic image and the sample image include the same foreground object, ensuring feature consistency for the foreground object. Using these sample images and synthetic images as image pairs for training the instance segmentation model improves the training effect. Because there is a significant neighborhood difference between the sample images and the synthetic images, this application only calculates image segmentation loss for the sample images and target classification loss and bounding box regression loss for the synthetic images. These losses are used to adjust and optimize the network parameters of the instance segmentation model, avoiding the impact of neighborhood bias on model training, thereby improving the recognition accuracy of the trained instance segmentation model and ultimately improving the accuracy of the instance segmentation results.

[0058] This application also proposes a two-stage training method for the instance segmentation model. In the first stage, the model is pre-trained using base class data to obtain a pre-trained instance segmentation model. In the second stage, the pre-trained model is trained using sample data with a small number of new classes to obtain a small-sample-trained instance segmentation model. Since base class data is common and readily available, while new class data is uncommon and difficult to obtain, by performing the first stage of pre-training with a large amount of readily available base class data, and then performing the second stage of fine-tuning training with a small amount of difficult-to-obtain new class data, the instance segmentation model obtained through this two-stage training possesses transfer learning capabilities, improving the recognition ability of the instance segmentation model and ensuring recognition accuracy.

[0059] Please see Figure 4 This figure is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application, applied to a server (referring to...). Figure 1 Taking server 102 as an example, the method may include the following steps: S401. Obtain sample data, including sample images and synthetic images.

[0060] In this embodiment, the instance segmentation model is trained by the server using sample data, which includes sample images, synthetic images, annotation information of the sample images, and annotation information of the synthetic images. The synthetic image is obtained by combining a foreground image extracted from the sample images and a portion of the background image. The foreground image includes the foreground objects in the sample images, ensuring that both the sample images and the synthetic images include foreground objects, thus achieving feature consistency between the sample images and the synthetic images for the foreground objects.

[0061] Since the foreground category of the foreground object in the sample image is the target foreground category, which is also the foreground category corresponding to the new data, and the target foreground category is a foreground category that is not common in real life and is not easily obtained, this application generates synthetic images from sample images, thereby increasing the sample dataset and improving data reusability. At the same time, the sample images and synthetic images have consistent features for the foreground object, and using the sample images and synthetic images as training sample groups for the instance segmentation model ensures the model training effect.

[0062] In one embodiment, the synthetic image in the sample data can be obtained by performing graphic transformation processing on the sample image. The process of generating a synthetic image from the sample image can be implemented according to the following steps: (a1) Obtain sample images and extract foreground images and part of the background images from the sample images.

[0063] In this embodiment, the server first acquires a sample image, and then extracts a foreground image from the sample image. The foreground image is the foreground region in the sample image. For example, if the background region of an image is a lawn, the goat on the lawn is the foreground object, and the image region corresponding to the goat is the foreground image. The server then extracts a partial background image from the sample image. This partial background image can be a portion of the background region of the sample image, such as a portion of the lawn in the image region corresponding to the lawn.

[0064] In one embodiment, a partial background image may not include the foreground image corresponding to the foreground object, or it may include the foreground image corresponding to the foreground object. Since the instance segmentation model will treat the partial background image extracted from the sample image as background by default during training, if the proportion of the foreground image corresponding to the foreground object in the partial background image is large, it will mislead the training of the instance segmentation model, reducing its training effect and recognition accuracy. Therefore, the server can limit the proportion of the foreground image corresponding to the foreground object included in the partial background image by setting a proportion threshold. For example, setting the proportion threshold to 1 / 8 means that the proportion of the foreground image corresponding to the foreground object included in the partial background image should not exceed 1 / 8. This method improves the distinguishability of the sample data between foreground and background, thereby improving the recognition accuracy of the instance segmentation model.

[0065] (a2) According to the image size of the sample image, scale a portion of the background image to obtain the scaled background image.

[0066] (a3) Perform image compositing on the foreground image and the scaled background image to obtain a composite image.

[0067] In the above steps (a2-a3), since part of the background image is a portion of the background region of the sample image, and the image size of part of the background image does not match that of the sample image, the server performs scaling processing on part of the background image, scaling it to the same image size as the sample image. This allows for efficient synthesis of a composite image that matches the image size of the sample image during subsequent image compositing of the foreground image and the scaled background image, thereby improving the generation efficiency of the composite image.

[0068] In one embodiment, the above-described image compositing process, which combines the foreground image and the scaled background image to obtain a composite image, can be implemented according to the following steps: (a31) Determine the position of the foreground image in the sample image.

[0069] (a32) Based on the position of the foreground image in the sample image, perform image compositing on the foreground image and the scaled background image to obtain the composite image.

[0070] In steps (a2-a3) above, the server performs image compositing on the foreground image and the scaled background image according to the position of the foreground image in the sample image, resulting in a composite image. At this point, the position of the foreground image in the composite image matches the position of the foreground image in the sample image. This method ensures that the annotation information for the foreground image in the composite image is consistent with the annotation information for the foreground image in the sample image. Therefore, annotators only need to annotate the sample data (including the reference foreground mask, reference foreground category, and reference foreground position bounding box) and then directly determine the annotation information of the composite data using the annotation information of the sample data, thereby improving data annotation efficiency.

[0071] Through the steps (a1-a3) and (a31-a32) described above, the relative position and size of the foreground object in the synthesized image are perfectly matched to the relative position and size of the foreground object in the sample image. Therefore, the annotation information of the synthesized image is consistent with the annotation information of the sample image, because both annotations are applied to their respective foreground objects. Based on this, the method for generating sample data can be implemented according to the following steps: (b1) Determine the annotation information of the sample image. The annotation information includes the reference foreground mask, reference foreground category and reference foreground location bounding box of the foreground object in the sample image.

[0072] (b2) The reference foreground category and reference foreground position bounding box of the foreground object in the sample image are determined as the reference foreground category and reference foreground position bounding box of the foreground object in the composite image.

[0073] In steps (b1-b2) above, since the annotation information of the synthesized image for the foreground image is consistent with that of the sample image for the foreground image, the server can use the reference foreground mask of the foreground object in the sample image as the reference foreground mask of the foreground object in the synthesized image; use the reference foreground category of the foreground object in the sample image as the reference foreground category of the foreground object in the synthesized image; and use the reference foreground position bounding box of the foreground object in the sample image as the reference foreground position bounding box of the foreground object in the synthesized image. This method ensures that the annotation information of the synthesized image and the sample image is completely consistent, improving the overall annotation efficiency of the sample data.

[0074] (b3) Generate sample data based on the sample image, the composite image, the annotation information of the sample image, the reference foreground category of the foreground object in the composite image, and the reference foreground position marker box.

[0075] In this embodiment, since the synthesized image is applied to the mask branch network, and the mask branch network is trained by the target classification loss (determined based on the predicted foreground category and the reference foreground category) and the bounding box regression loss (determined based on the predicted foreground location bounding box and the reference foreground location bounding box), this embodiment uses the reference foreground category and the reference foreground location bounding box of the foreground object in the synthesized image as the annotation information of the synthesized image.

[0076] In one embodiment, the annotation information of the sample data is divided into annotation information for the sample image and annotation information for the synthesized image. The annotation information for the sample image may include a reference foreground mask, a reference foreground category, and a reference foreground location bounding box for the foreground object in the sample image; the annotation information for the synthesized image may include a reference foreground mask, a reference foreground category, and a reference foreground location bounding box for the foreground object in the synthesized image. The annotation information of the sample image and the annotation information of the synthesized image are a perfect match; that is, the reference foreground mask of the foreground object in the sample image matches the reference foreground mask of the foreground object in the synthesized image; the reference foreground category of the foreground object in the sample image matches the reference foreground category of the foreground object in the synthesized image; and the reference foreground location bounding box of the foreground object in the sample image matches the reference foreground location bounding box of the foreground object in the synthesized image.

[0077] It should be noted that the relative position and size of the foreground object in the synthesized image match the relative position and size of the foreground object in the sample image, ensuring complete consistency of the annotation information between the two images. Annotators only need to annotate either the synthesized image or the sample image, improving annotation efficiency. However, considering actual business environments, the relative position and size of the foreground object in the synthesized image may not match the relative position and size of the foreground object in the sample image. In this case, annotators need to annotate both the synthesized image and the sample image separately, improving scenario applicability.

[0078] The process of generating a synthetic image will be introduced below through a specific example, given a sample image. (i.e., small sample labeled images) and their corresponding annotation information ,in , , These represent the sample image number 1 and 2 respectively. The annotation information for each instance (i.e., the foreground object) includes the foreground location bounding box, foreground mask, and foreground category. Generating a simple background image (i.e., a composite image) from a sample image mainly involves the following three steps: (1) Considering the first Foreground mask for each instance Since this is known, the foreground object image of the pure background of this instance can be obtained. ,in, This indicates a multiplication operation involving corresponding elements.

[0079] (2) Randomly crop a small background image patch from the sample image, and then scale it to the size of the sample image. Let the scaled image patch be denoted as . .

[0080] (3) Image of foreground object with pure background Image patches scaled after random cropping Creating a new image by combining elements is called image synthesis. Compared to the sample image, the background of the synthesized image is simpler. The process of synthesizing the image can be expressed by the formula:

[0081] The process of generating the composite image will be illustrated below. Please refer to [link / reference]. Figure 5AThis figure is a flowchart illustrating a method for generating a composite image, provided in an exemplary embodiment of this application. The server randomly crops a portion of the background image block from the sample image, and scales the background image block to the size of the original image (i.e., the sample image) to obtain the background image. The server then extracts the foreground image corresponding to the foreground object (e.g., the cat in the sample image) from the sample image, and performs image compositing processing on the foreground and background images to obtain the composite image.

[0082] In one embodiment, the server can define the rules for random cropping. For example, the random cropping of background image patches should, as far as possible, satisfy the following principles: on the one hand, it should ensure that the background image patches of the synthesized image contain rich contextual information (e.g., background image patches with color distribution can be collected, avoiding the collection of solid-color background image patches); on the other hand, it should minimize the probability of potential foreground objects appearing in the background image patches (e.g., background image patches other than potential foreground objects can be collected). Assume that the size (i.e., height and width) of the randomly cropped background image patch is a fraction of the sample image size (i.e., height and width). ( (Multiple). In practical applications, in order to satisfy the aforementioned principles as much as possible, Values ​​in Generally, better performance can be achieved. Compared to using the original background of the sample image, the probability of a cropped background image patch containing a hidden foreground object is much lower. Even if the background image patch happens to contain an unlabeled target, a large scaling ratio will make the target look completely different. By using the above method, the probability of potential foreground objects appearing in the background region of the synthetic image is reduced. Using this synthetic data and sample data as sample pairs to train the instance segmentation model improves the recognition accuracy of the instance segmentation model.

[0083] Based on the above process for generating synthetic images, each given a small sample labeled image... This allows you to create an image with a simple background. Based on the above synthesis principles, the two resulting images have completely identical annotation information, meaning they are both... For ease of description later, this application represents the sample image and the synthesized image as an image pair, denoted as... .

[0084] The following is a pseudocode demonstration of generating composite images in PyTorch style:

[0085] in, The function is used to return random values. The floating-point numbers between these values, where a can take the value 0.125 and b can take the value 0.25. Functions are used to base arguments on parameters A portion of the background image is randomly cropped from the sample image. Used for parameter-based For image patches The dimensions were adjusted.

[0086] Due to the influence of neighborhood offset, simply training instance segmentation models on synthetic images in a traditional manner yields poor results. This is because sample images and synthetic images have very different content, making it difficult for instance segmentation models trained on synthetic data to generalize well to instance segmentation tasks on real images. Based on this, this application proposes a simple and effective training method to alleviate the neighborhood offset problem. The training method of the instance segmentation model in this application will be described below: S402. Perform feature extraction processing on the sample image to determine the first global feature map of the sample image, and determine the first candidate region feature map based on the first global feature map.

[0087] In this embodiment, the image region corresponding to the first candidate region feature map includes the foreground object in the sample image. The server can perform feature extraction processing on the sample image to obtain a first global feature map that characterizes the overall image features of the sample image. This allows the server to determine the first candidate region feature map corresponding to the foreground object in the sample image based on the first global feature map, thereby completing the image feature extraction steps in the two stages described above and improving the accuracy of feature extraction.

[0088] S403. Perform feature extraction processing on the synthesized image to determine the second global feature map of the synthesized image, and determine the second candidate region feature map based on the second global feature map.

[0089] In this embodiment, the image region corresponding to the second candidate region feature map includes the foreground object in the synthesized image. The server can perform feature extraction processing on the synthesized image to obtain a second global feature map that characterizes the overall image features of the synthesized image. This allows the server to determine the second candidate region feature map corresponding to the foreground object in the synthesized image based on the second global feature map, thereby realizing the image feature extraction steps in the two stages described above and improving the accuracy of feature extraction.

[0090] In one embodiment, the steps of performing feature extraction processing on the sample image and the synthetic image can be performed by the server through a feature extraction network, which can be a sub-network of the instance segmentation model proposed in this application.

[0091] Taking the determination of the second candidate region feature map based on the second global feature map of the synthesized image as an example, the above steps can be implemented in the following way: the server inputs the second global feature map into the candidate region generation network for processing to determine the second candidate region feature map.

[0092] The candidate region generation network is trained using candidate region regression loss and candidate region classification loss obtained from the synthesized image. The candidate region generation network generates candidate region bounding boxes corresponding to foreground objects, while the annotation information of the synthesized image includes reference foreground location bounding boxes for the foreground objects. The candidate region regression loss is determined by calculating the losses of the candidate region bounding boxes and the reference foreground location bounding boxes of the synthesized image. Similarly, the candidate region generation network generates candidate region categories for the candidate region bounding boxes corresponding to foreground objects, while the annotation information of the synthesized image includes reference foreground category for the reference foreground location bounding boxes corresponding to the foreground objects. The candidate region classification loss is determined by calculating the losses of the candidate region categories and the reference foreground category of the synthesized image. By adjusting the network parameters of the candidate region generation network using the candidate region regression loss and candidate region classification loss, the trained candidate region generation network is obtained.

[0093] It should be noted that this application includes two types of classification losses: the object classification loss corresponding to the object detection branch network, and the candidate region classification loss corresponding to the candidate region generation network. The object classification loss is calculated based on the predicted foreground category and the reference foreground category, which include a variety of preset foreground categories, such as cow, horse, cat, sunflower, and goat. The candidate region classification loss is calculated based on the candidate region category and the reference foreground category, which includes foreground and background categories. The server needs to convert the reference foreground category into a reference region category (e.g., if the reference foreground category is cow, then the candidate region category is foreground), and calculate the candidate region classification loss together with the candidate region category.

[0094] In one embodiment, the candidate region generation network can employ a Region Proposal Network (RPN) and be implemented through the ROI-Align module. The RPN is a processing network in the Faster R-CNN model (an object detection model) used to extract candidate boxes corresponding to foreground objects in an image. The RPN can be understood as performing a first-stage candidate box (proposal) generation process on the image, providing high-quality candidate boxes for the ROI-Align module. The ROI-Align module can be understood as, based on the candidate boxes provided by the RPN, selecting training samples for the second stage and extracting corresponding features to build the second-stage training network. Simply put, the ROI-Align module optimizes the candidate boxes from the first stage using interpolation algorithms, upsampling, and forward computation, thereby obtaining candidate boxes that better match the target objects in the image.

[0095] It should be noted that the determination of the first candidate region feature map based on the first global feature map of the sample image and the determination of the target candidate region feature map based on the target global feature map of the image to be processed in this application can both be implemented based on the above-mentioned synthetic image processing method, and will not be elaborated further in this application.

[0096] S404. Input the feature map of the first candidate region into the mask branch network for processing, determine the predicted foreground mask of the foreground object in the sample image, determine the image segmentation loss based on the predicted foreground mask and the reference foreground mask, and adjust the network parameters of the mask branch network based on the image segmentation loss to obtain the trained mask branch network.

[0097] In this embodiment, the server uses a mask branch network to identify the feature map of the first candidate region, obtaining a predicted foreground mask for the foreground object in the sample image. The reference foreground mask in the annotation information is used as supervision information for the predicted foreground mask in loss calculation to determine the image segmentation loss. The server then uses the image segmentation loss to adjust the network parameters of the mask branch network, resulting in a trained mask branch network.

[0098] S405. Input the feature map of the second candidate region into the target detection branch network for processing, determine the predicted foreground category and predicted foreground location bounding box of the foreground object in the synthetic image, determine the target classification loss based on the predicted foreground category and the reference foreground category, determine the bounding box regression loss based on the predicted foreground location bounding box and the reference foreground location bounding box, and adjust the network parameters of the target detection branch network based on the target classification loss and the bounding box regression loss to obtain the trained target detection branch network.

[0099] In this embodiment, the server calls the object detection branch network to perform recognition processing on the feature map of the second candidate region, obtaining the predicted foreground category and predicted foreground location bounding box of the foreground object in the sample image. The reference foreground category in the annotation information is used as the supervision information for the predicted foreground category to calculate the loss, determining the object classification loss; the reference foreground location bounding box in the annotation information is used as the supervision information for the predicted foreground location bounding box to calculate the loss, determining the bounding box regression loss; the server then uses the object classification loss and the bounding box regression loss to adjust the network parameters of the object detection branch network, obtaining the trained object detection branch network.

[0100] In this application, the object detection branch network and the masking branch network can adopt the mainstream MASK R-CNN structure (a two-stage instance segmentation model that can determine the position and category of each foreground object in the image and give pixel-level prediction results), which will not be described in detail in this embodiment.

[0101] S406. Construct an instance segmentation model based on the trained masking branch network and the trained object detection branch network.

[0102] In this embodiment, the server constructs an instance segmentation model using a trained masking branch network and a trained object detection branch network. In addition, the instance segmentation model also includes a candidate region generation network.

[0103] The training process of the generated instance segmentation model will be illustrated below. Please refer to [link / reference]. Figure 5B This figure is a training flowchart of an instance segmentation model based on image pairs provided in an exemplary embodiment of this application. The image pairs (i.e., sample data) include sample images and synthesized images. The sample images and synthesized images are respectively input into a convolutional neural network (Backbone network) to obtain a first global feature map of the sample images and a second global feature map of the synthesized images. The first global feature map is processed by a candidate region generation network (e.g., an RPN network) and a ROI-Align module to obtain a first candidate region feature map corresponding to the foreground object in the sample image. Similarly, the second global feature map is processed by a candidate region generation network (e.g., an RPN network) and a ROI-Align module to obtain a second candidate region feature map corresponding to the foreground object in the synthesized image.

[0104] In the sample image branch, the server inputs the first candidate region feature map into the mask branch network for processing to determine the predicted foreground mask of the foreground object in the sample image. The image segmentation loss is then determined based on the predicted foreground mask and the reference foreground mask. In the synthetic image branch, the server inputs the second candidate region feature map into the object detection branch network for processing to determine the predicted foreground category and predicted foreground location bounding box of the foreground object in the synthetic image. The object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding box and the reference foreground location bounding box. Simultaneously, the server trains on the candidate region regression loss and candidate region classification loss obtained from the synthetic image. Specifically, the candidate region generation network generates candidate region bounding boxes corresponding to the foreground object, while the annotation information of the synthetic image includes the reference foreground location bounding box corresponding to the foreground object. By calculating the loss between the candidate region bounding boxes and the reference foreground location bounding box of the synthetic image, the candidate region regression loss can be determined. Similarly, the candidate region generation network can generate candidate region categories for candidate region bounding boxes corresponding to foreground objects, while the annotation information of the synthesized image includes the reference foreground category of the reference foreground location bounding box corresponding to the foreground object. By calculating the loss between the candidate region categories and the reference foreground category of the synthesized image, the candidate region classification loss can be determined.

[0105] The server then adjusts the network parameters of the mask branch network using image segmentation loss to obtain the trained mask branch network; adjusts the network parameters of the object detection branch network using object classification loss and bounding box regression loss to obtain the trained object detection branch network; adjusts the network parameters of the candidate region generation network using candidate region regression loss and candidate region classification loss to obtain the trained candidate region generation network, and finally constructs an instance segmentation model based on the above three losses.

[0106] To reduce the domain gap between sample images and synthetic images, this application proposes to compute only image segmentation loss on sample images, while computing object classification loss and bounding box regression loss on synthetic images. Assuming that the sample images and synthetic images undergo forward computation through the Backbone, RPN network, and ROI-Align module, the resulting depth features are as follows: and .Then, and The data are fed into the Mask Head and Box Head networks, respectively. Finally, the Mask Head uses an image segmentation loss function to learn pixel-level instance segmentation, while the Box Head uses a bounding box-based object classification loss function and a bounding box regression loss function to learn instance detection.

[0107] The masking branch network encourages the instance segmentation model to separate background and foreground pixels to compute the loss given a good bounding box, which prevents the model from suppressing the detection of potential objects. Furthermore, the training signal for the image segmentation loss is backpropagated to the backbone network shared between the RPN network, the bounding box head, and the masking branch, thus promoting the adaptation of the bounding box head to the sample image. Since the detection box head and the RPN network are trained using only synthetic images with simple backgrounds, they are better at detecting novel objects (i.e., they have transfer learning capabilities), thereby improving the performance of few-shot instance segmentation.

[0108] In the inference phase of the instance segmentation model in this application, the model only needs to input the image to be processed, without synthesizing an image. The main difference compared to the pairwise training process lies in the object detection branch. Since the inference phase only involves the image to be processed, only the input from the object detection branch network in the pairwise training phase needs to be used. Adjust to the image to be processed That's all.

[0109] S407. Perform feature extraction processing on the image to be processed, determine the target global feature map of the image to be processed, and determine the target candidate region feature map based on the target global feature map.

[0110] The image region corresponding to the target candidate region feature map includes the foreground object in the image to be processed.

[0111] S408. Input the target candidate region feature map into the mask branch network in the instance segmentation model for processing to determine the foreground mask corresponding to the foreground object, and input the target candidate region feature map into the target detection branch network in the instance segmentation model for processing to determine the foreground category and foreground location bounding box corresponding to the foreground object.

[0112] S409. Determine the instance segmentation result of the image to be processed based on the foreground mask, foreground category, and foreground position bounding box corresponding to the foreground object.

[0113] The specific implementation methods of steps S407-S409 are described in the relevant descriptions of steps S201-S205 in the foregoing embodiments, and will not be repeated here.

[0114] Based on the above embodiments, the beneficial effects of this application are as follows: This application proposes to scale a portion of the background image to the same image size as the sample image. This allows for efficient synthesis of a composite image matching the image size of the sample image during subsequent image compositing of the foreground image and the scaled background image, thereby improving the generation efficiency of the composite image. By performing image compositing on the foreground image and the scaled background image according to the position of the foreground image in the sample image, it is ensured that the annotation information of the composite image and the sample image is completely consistent. Annotators only need to annotate the sample data, improving the overall annotation efficiency of the sample data.

[0115] This application also proposes to limit the rules for random cropping, on the one hand, to ensure that the background image patch of the synthesized image contains rich contextual information as much as possible, and on the other hand, to reduce the probability of potential foreground targets appearing in the background image patch as much as possible. By using the above method, the probability of potential foreground targets appearing in the background region of the synthesized image is reduced. Using this synthesized data and sample data as sample pairs for training the instance segmentation model improves the recognition accuracy of the instance segmentation model.

[0116] The performance test results of the instance segmentation model in this application will be presented below, based on experimental data: MS-COCO is a highly challenging benchmark dataset commonly used in few-shot detection and instance segmentation, involving 80 base class categories. Following the general setup for few-shot instance segmentation, this application uses 5000 images from the validation set for performance evaluation, and the remaining validation and training set images are used to train the instance segmentation model. Adhering to standard evaluation protocols, this application uses the same 20 categories as PASCAL VOC (a standardized dataset for image recognition and classification) as new classes, and the remaining 60 categories as base classes. Each new class has only one instance labeled, i.e., 1-shot.

[0117] Similar to MTFA (a superior few-shot instance segmentation method), this application compares performance in 1-shot mode based on a more generalized few-shot instance evaluation metric. This metric reports not only the evaluation metrics for the new class but also the evaluation metrics for the base class, as well as the average evaluation metrics across all classes. For all evaluation metrics, base class evaluation metrics, and new class evaluation metrics, both AP and AP50 for instance detection and segmentation are provided, and their performance is compared with the current best few-shot instance segmentation method, MTFA. The specific comparison results are shown in Tables 1 and 2 below.

[0118]

[0119] Table 1 Among them, the instance detection metric refers to the metric corresponding to the predicted foreground location bounding box and the reference foreground location bounding box. AP (Average Precision) refers to the average recognition precision at a certain recall rate. AP50 refers to the average recognition precision at a recall rate of 50, which is the average recognition precision when the Intersection Over Union (IOU) between the predicted foreground location bounding box and the reference foreground location bounding box is 50%.

[0120]

[0121] Table 2 Among them, the instance segmentation metric refers to the metric corresponding to the predicted foreground mask and the reference foreground mask. AP (Average Precision) refers to the average recognition precision at a certain recall rate. AP50 refers to the average recognition precision at a recall rate of 50, which is the average recognition precision when the Intersection Over Union (IOU) between the predicted foreground mask and the reference foreground mask is 50%.

[0122] As can be seen from Tables 1 and 2 above, the method proposed in this application shows a significant improvement over the current best MTFA in both instance detection and instance segmentation metrics, especially in the detection and segmentation of new classes, where there is an improvement of 1-1.5 points, which verifies the effectiveness of the method in this application.

[0123] Furthermore, the method proposed in this application can be directly applied to the field of industrial quality inspection, such as the inspection of appearance defects in LCD panels, 3C products, and new energy batteries. This is because defect judgment in these industries involves not only the category but also the precise relationship between the location and size of the defect, making instance segmentation algorithms highly suitable. Considering the difficulty in collecting defect data in these inspection scenarios, the small-sample instance segmentation technique proposed in this application can effectively improve the performance of defect detection and segmentation in small-sample scenarios.

[0124] Please see Figure 6 This figure is a schematic block diagram of an image processing apparatus provided in an embodiment of this application. Specifically, the image processing apparatus may include: Preprocessing module 601 is used to perform feature extraction processing on the image to be processed and determine the target global feature map of the image to be processed. The preprocessing module 601 is further configured to determine a target candidate region feature map based on the target global feature map, wherein the image region corresponding to the target candidate region feature map includes the foreground object in the image to be processed. The instance segmentation module 602 is used to input the above target candidate region feature map into the mask branch network in the instance segmentation model for processing, and to determine the foreground mask corresponding to the above foreground object. The instance segmentation module 602 described above is also used to input the feature map of the target candidate region into the target detection branch network in the instance segmentation model for processing, and to determine the foreground category and foreground location bounding box corresponding to the foreground object. The output module 603 is used to determine the instance segmentation result of the image to be processed based on the foreground mask, foreground category and foreground position bounding box corresponding to the foreground object. The instance segmentation model is trained using sample data, which includes sample images, synthesized images, a reference foreground mask for the sample images, a reference foreground category for the synthesized images, and reference foreground location bounding boxes. The synthesized images are obtained by combining foreground images extracted from the sample images with a portion of the background image. The foreground images include foreground objects from the sample images, and the foreground category of the foreground objects in the sample images is the target foreground category. The mask branch network is trained using an image segmentation loss, which is determined based on the predicted foreground mask and the reference foreground mask. The predicted foreground mask is obtained by processing the candidate region feature maps of the sample images using the mask branch network. The object detection branch network is trained using an object classification loss and a bounding box regression loss, whereby the object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding boxes and the reference foreground location bounding boxes. The predicted foreground category and the predicted foreground location bounding boxes are obtained by processing the candidate region feature maps of the synthesized images using the object detection branch network.

[0125] Optionally, the preprocessing module 601 described above is also used for: The sample data obtained includes the sample images and the synthesized images. The instance segmentation module 602 described above is also used for: Feature extraction processing is performed on the above sample images to determine the first global feature map of the above sample images. Based on the first global feature map, a first candidate region feature map is determined. The image region corresponding to the first candidate region feature map includes the foreground object in the above sample images. The above-mentioned synthesized image is subjected to feature extraction processing to determine the second global feature map of the above-mentioned synthesized image. Based on the second global feature map, a second candidate region feature map is determined. The image region corresponding to the second candidate region feature map includes the foreground object in the above-mentioned synthesized image. The first candidate region feature map is input into the mask branch network for processing to determine the predicted foreground mask of the foreground object in the sample image. The image segmentation loss is determined based on the predicted foreground mask and the reference foreground mask. The network parameters of the mask branch network are adjusted based on the image segmentation loss to obtain the trained mask branch network. The second candidate region feature map is input into the target detection branch network for processing to determine the predicted foreground category and predicted foreground location bounding box of the foreground object in the synthesized image. The target classification loss is determined based on the predicted foreground category and the reference foreground category. The bounding box regression loss is determined based on the predicted foreground location bounding box and the reference foreground location bounding box. The network parameters of the target detection branch network are adjusted based on the target classification loss and the bounding box regression loss to obtain the trained target detection branch network. The instance segmentation model described above is constructed based on the trained masking branch network and the trained object detection branch network described above.

[0126] Optionally, the preprocessing module 601 described above is also used for: Obtain the above sample images, and extract the foreground image and part of the background image from the above sample images; Based on the image size of the above sample image, the above-mentioned background image is scaled to obtain the scaled background image; The foreground image and the scaled background image are combined to obtain the composite image.

[0127] Optionally, the preprocessing module 601 described above is also used for: Determine the position of the foreground image in the sample image. Based on the position of the foreground image in the sample image, the foreground image and the scaled background image are combined to obtain the composite image.

[0128] Optionally, the preprocessing module 601 described above is also used for: The annotation information of the above sample image is determined, including the reference foreground mask, reference foreground category, and reference foreground location bounding box of the foreground object in the above sample image; The reference foreground category and reference foreground position bounding box of the foreground object in the above sample image are determined as the reference foreground category and reference foreground position bounding box of the foreground object in the above composite image; The above sample data is generated based on the above sample image, the above composite image, the annotation information of the above sample image, the reference foreground category of the foreground object in the above composite image, and the reference foreground position marker box.

[0129] Optionally, the instance segmentation model described above further includes a candidate region generation network. When the instance segmentation module 602 determines the target candidate region feature map based on the target global feature map, it is specifically used for: The target global feature map is input into the candidate region generation network for processing to determine the target candidate region feature map. The candidate region generation network is trained based on candidate region regression loss and candidate region classification loss. The candidate region regression loss is determined based on the candidate region bounding box and the reference foreground location bounding box of the synthesized image. The candidate region classification loss is determined based on the candidate region category and the reference foreground category of the synthesized image. The candidate region bounding box and the candidate region category are obtained by processing the global feature map of the synthesized image through the candidate region generation network.

[0130] Optionally, the preprocessing module 601 described above is also used for: The instance segmentation model is initially trained using the base class data to obtain the pre-trained instance segmentation model; the base class data includes base class images, and the foreground category of the foreground object in the base class images is different from the foreground category of the foreground object in the sample images. The instance segmentation model trained using the aforementioned sample data is the instance segmentation model trained in the preliminary stage.

[0131] Based on the above image processing apparatus, this application extracts a global feature map of the image to be processed, and then extracts a candidate region feature map corresponding to the foreground object in the image to be processed based on the global feature map. The image feature extraction is completed through the above two stages, thereby improving the accuracy of feature extraction.

[0132] By using a masking branch network to determine the foreground mask of the foreground object, and an object detection branch network to determine the foreground category and foreground location bounding box of the foreground object, the instance segmentation results obtained by the instance segmentation model contain multi-dimensional feature information, improving the data richness of the recognition results and thus achieving more refined recognition.

[0133] During the training phase of the instance segmentation model, the model is trained using sample images and synthetic images. Since sample images are a target type that is difficult to obtain, synthetic images are synthesized by extracting foreground images and part of the background images from the sample images, which increases the sample dataset and improves the data reuse rate.

[0134] The fact that the synthetic image and the sample image both contain the same foreground object ensures that the sample image and the synthetic image have consistent features for the foreground object. Using the above sample image and synthetic image as image pairs to train the instance segmentation model can improve the model training effect.

[0135] Considering the significant domain difference between sample images and synthetic images, this application calculates only image segmentation loss for sample images and target classification loss and bounding box regression loss for synthetic images. By adjusting and optimizing the network parameters of the instance segmentation model through the above losses, the influence of domain offset on model training is avoided, thereby improving the recognition accuracy of the trained instance segmentation model and thus improving the accuracy of instance segmentation results.

[0136] It should be noted that the functions of each functional module of the image processing device in this application embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0137] Please see Figure 7 This figure is a schematic block diagram of a computer device provided in an embodiment of this application. As shown in the figure, the smart terminal in this embodiment may include: a processor 701, a storage device 702, and a communication interface 703. The processor 701, storage device 702, and communication interface 703 can interact with each other.

[0138] The aforementioned storage device 702 may include volatile memory, such as random-access memory (RAM); the storage device 702 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the aforementioned storage device 702 may also include a combination of the above types of memory.

[0139] The processor 701 described above may be a central processing unit (CPU). In one embodiment, the processor 701 may also be a graphics processing unit (GPU). Alternatively, the processor 701 may be a combination of a CPU and a GPU. In one embodiment, the storage device 702 is used to store program instructions, and the processor 701 can invoke these program instructions to perform the following operations: Feature extraction is performed on the image to be processed to determine the target global feature map of the image to be processed. Based on the above target global feature map, a target candidate region feature map is determined. The image region corresponding to the above target candidate region feature map includes the foreground object in the above image to be processed. The above target candidate region feature map is input into the mask branch network in the instance segmentation model for processing to determine the foreground mask corresponding to the above foreground object. The above target candidate region feature map is input into the target detection branch network in the above instance segmentation model for processing to determine the foreground category and foreground location bounding box corresponding to the above foreground object; Based on the foreground mask, foreground category, and foreground location bounding box corresponding to the foreground object, the instance segmentation result of the image to be processed is determined. The instance segmentation model is trained using sample data, which includes sample images, synthesized images, a reference foreground mask for the sample images, a reference foreground category for the synthesized images, and reference foreground location bounding boxes. The synthesized images are obtained by combining foreground images extracted from the sample images with a portion of the background image. The foreground images include foreground objects from the sample images, and the foreground category of the foreground objects in the sample images is the target foreground category. The mask branch network is trained using an image segmentation loss, which is determined based on the predicted foreground mask and the reference foreground mask. The predicted foreground mask is obtained by processing the candidate region feature maps of the sample images using the mask branch network. The object detection branch network is trained using an object classification loss and a bounding box regression loss, whereby the object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding boxes and the reference foreground location bounding boxes. The predicted foreground category and the predicted foreground location bounding boxes are obtained by processing the candidate region feature maps of the synthesized images using the object detection branch network.

[0140] Optionally, the processor 701 described above is also used for: The sample data obtained includes the sample images and the synthesized images. Feature extraction processing is performed on the above sample images to determine the first global feature map of the above sample images. Based on the first global feature map, a first candidate region feature map is determined. The image region corresponding to the first candidate region feature map includes the foreground object in the above sample images. The above-mentioned synthesized image is subjected to feature extraction processing to determine the second global feature map of the above-mentioned synthesized image. Based on the second global feature map, a second candidate region feature map is determined. The image region corresponding to the second candidate region feature map includes the foreground object in the above-mentioned synthesized image. The first candidate region feature map is input into the mask branch network for processing to determine the predicted foreground mask of the foreground object in the sample image. The image segmentation loss is determined based on the predicted foreground mask and the reference foreground mask. The network parameters of the mask branch network are adjusted based on the image segmentation loss to obtain the trained mask branch network. The second candidate region feature map is input into the target detection branch network for processing to determine the predicted foreground category and predicted foreground location bounding box of the foreground object in the synthesized image. The target classification loss is determined based on the predicted foreground category and the reference foreground category. The bounding box regression loss is determined based on the predicted foreground location bounding box and the reference foreground location bounding box. The network parameters of the target detection branch network are adjusted based on the target classification loss and the bounding box regression loss to obtain the trained target detection branch network. The instance segmentation model described above is constructed based on the trained mask branch network and the trained object detection branch network described above.

[0141] Optionally, the processor 701 described above is also used for: Obtain the above sample images, and extract the foreground image and part of the background image from the above sample images; Based on the image size of the above sample image, the above-mentioned background image is scaled to obtain the scaled background image; The foreground image and the scaled background image are combined to obtain the composite image.

[0142] Optionally, the processor 701 described above is also used for: Determine the position of the foreground image in the sample image. Based on the position of the foreground image in the sample image, the foreground image and the scaled background image are combined to obtain the composite image.

[0143] Optionally, the processor 701 described above is also used for: The annotation information of the above sample image is determined, including the reference foreground mask, reference foreground category, and reference foreground location bounding box of the foreground object in the above sample image; The reference foreground category and reference foreground position bounding box of the foreground object in the above sample image are determined as the reference foreground category and reference foreground position bounding box of the foreground object in the above composite image; The above sample data is generated based on the above sample image, the above composite image, the annotation information of the above sample image, the reference foreground category of the foreground object in the above composite image, and the reference foreground position marker box.

[0144] Optionally, the instance segmentation model described above further includes a candidate region generation network. When the processor 701 determines the target candidate region feature map based on the target global feature map, it specifically performs the following functions: The target global feature map is input into the candidate region generation network for processing to determine the target candidate region feature map. The candidate region generation network is trained based on candidate region regression loss and candidate region classification loss. The candidate region regression loss is determined based on the candidate region bounding box and the reference foreground location bounding box of the synthesized image. The candidate region classification loss is determined based on the candidate region category and the reference foreground category of the synthesized image. The candidate region bounding box and the candidate region category are obtained by processing the global feature map of the synthesized image through the candidate region generation network.

[0145] Optionally, the processor 701 described above is also used for: The instance segmentation model is initially trained using the base class data to obtain the pre-trained instance segmentation model; the base class data includes base class images, and the foreground category of the foreground object in the base class images is different from the foreground category of the foreground object in the sample images. The instance segmentation model trained using the aforementioned sample data is the instance segmentation model trained in the preliminary stage.

[0146] Based on the aforementioned computer equipment, this application extracts a global feature map of the image to be processed, and then extracts a candidate region feature map corresponding to the foreground object in the image to be processed based on the global feature map. The image feature extraction is completed through the above two stages, thereby improving the accuracy of feature extraction.

[0147] By using a masking branch network to determine the foreground mask of the foreground object, and an object detection branch network to determine the foreground category and foreground location bounding box of the foreground object, the instance segmentation results obtained by the instance segmentation model contain multi-dimensional feature information, improving the data richness of the recognition results and thus achieving more refined recognition.

[0148] During the training phase of the instance segmentation model, the model is trained using sample images and synthetic images. Since sample images are a target type that is difficult to obtain, synthetic images are synthesized by extracting foreground images and part of the background images from the sample images, which increases the sample dataset and improves the data reuse rate.

[0149] The fact that the synthetic image and the sample image both contain the same foreground object ensures that the sample image and the synthetic image have consistent features for the foreground object. Using the above sample image and synthetic image as image pairs to train the instance segmentation model can improve the model training effect.

[0150] Considering the significant domain difference between sample images and synthetic images, this application calculates only image segmentation loss for sample images and target classification loss and bounding box regression loss for synthetic images. By adjusting and optimizing the network parameters of the instance segmentation model through the above losses, the influence of domain offset on model training is avoided, thereby improving the recognition accuracy of the trained instance segmentation model and thus improving the accuracy of instance segmentation results.

[0151] In specific implementation, the processor 701, storage device 702, and communication interface 703 described in the embodiments of this application can execute the embodiments of this application. Figure 2 or Figure 4 The implementation methods described in the relevant embodiments of the provided image processing method can also be used to execute the embodiments of this application. Figure 6 The implementation methods described in the relevant embodiments of the provided image processing apparatus will not be repeated here.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0153] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium storing a computer program executed by the aforementioned image processing apparatus. This computer program includes program instructions, which, when executed by a processor, enable the execution of the aforementioned image processing apparatus. Figure 2 , Figure 4 The methods described in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0154] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 2 , Figure 4 The methods described in the corresponding embodiments are therefore not repeated here.

[0155] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0156] The above-disclosed embodiments are merely some of the embodiments of this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of the invention.

Claims

1. An image processing method, characterized in that, The method includes: Perform feature extraction processing on the image to be processed to determine the target global feature map of the image to be processed; A target candidate region feature map is determined based on the target global feature map, and the image region corresponding to the target candidate region feature map includes the foreground object in the image to be processed; The feature map of the target candidate region is input into the mask branch network in the instance segmentation model for processing to determine the foreground mask corresponding to the foreground object. The target candidate region feature map is input into the target detection branch network in the instance segmentation model for processing to determine the foreground category and foreground location bounding box corresponding to the foreground object; Based on the foreground mask, foreground category, and foreground location bounding box corresponding to the foreground object, the instance segmentation result of the image to be processed is determined; The instance segmentation model is trained using sample data, which includes sample images, synthesized images, a reference foreground mask for the sample images, a reference foreground category for the synthesized images, and reference foreground location bounding boxes. The synthesized images are obtained by combining foreground images extracted from the sample images and a portion of the background image. The foreground images include foreground objects in the sample images, and the foreground category of the foreground objects in the sample images is the target foreground category. The mask branch network is trained using an image segmentation loss, which is determined based on a predicted foreground mask and the reference foreground mask. The predicted foreground mask is obtained by processing the candidate region feature map of the sample images using the mask branch network. The object detection branch network is trained using an object classification loss and a bounding box regression loss, whereby the object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding boxes and the reference foreground location bounding boxes. The predicted foreground category and the predicted foreground location bounding boxes are obtained by processing the candidate region feature map of the synthesized images using the object detection branch network.

2. The method according to claim 1, characterized in that, The method further includes: The sample data includes the sample image and the synthesized image; The sample image is subjected to feature extraction processing to determine a first global feature map of the sample image, and a first candidate region feature map is determined based on the first global feature map. The image region corresponding to the first candidate region feature map includes the foreground object in the sample image. The synthesized image is subjected to feature extraction processing to determine a second global feature map of the synthesized image. A second candidate region feature map is determined based on the second global feature map. The image region corresponding to the second candidate region feature map includes the foreground object in the synthesized image. The first candidate region feature map is input into the mask branch network for processing to determine the predicted foreground mask of the foreground object in the sample image. The image segmentation loss is determined based on the predicted foreground mask and the reference foreground mask. The network parameters of the mask branch network are adjusted based on the image segmentation loss to obtain the trained mask branch network. The second candidate region feature map is input into the target detection branch network for processing to determine the predicted foreground category and predicted foreground location bounding box of the foreground object in the synthesized image. The target classification loss is determined based on the predicted foreground category and the reference foreground category. The bounding box regression loss is determined based on the predicted foreground location bounding box and the reference foreground location bounding box. The network parameters of the target detection branch network are adjusted based on the target classification loss and the bounding box regression loss to obtain the trained target detection branch network. The instance segmentation model is constructed based on the trained masking branch network and the trained object detection branch network.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Acquire the sample image, and extract the foreground image and a portion of the background image from the sample image; The background image is scaled according to the image size of the sample image to obtain a scaled background image; The foreground image and the scaled background image are combined to obtain the composite image.

4. The method according to claim 3, characterized in that, The step of performing image compositing on the foreground image and the scaled background image to obtain the composite image includes: Determine the position of the foreground image in the sample image; Based on the position of the foreground image in the sample image, the foreground image and the scaled background image are combined to obtain the composite image.

5. The method according to claim 4, characterized in that, The method further includes: The annotation information of the sample image is determined, and the annotation information includes the reference foreground mask, reference foreground category, and reference foreground position bounding box of the foreground object in the sample image; The reference foreground category and reference foreground position bounding box of the foreground object in the sample image are determined as the reference foreground category and reference foreground position bounding box of the foreground object in the synthesized image; The sample data is generated based on the sample image, the composite image, the annotation information of the sample image, the reference foreground category of the foreground object in the composite image, and the reference foreground position marker box.

6. The method according to claim 1 or 2, characterized in that, The instance segmentation model further includes a candidate region generation network, wherein determining the target candidate region feature map based on the target global feature map includes: The target global feature map is input into the candidate region generation network for processing to determine the target candidate region feature map; The candidate region generation network is trained based on candidate region regression loss and candidate region classification loss. The candidate region regression loss is determined based on the candidate region bounding box and the reference foreground position bounding box of the synthesized image. The candidate region classification loss is determined based on the candidate region category and the reference foreground category of the synthesized image. The candidate region bounding box and the candidate region category are obtained by processing the global feature map of the synthesized image through the candidate region generation network.

7. The method according to claim 1 or 2, characterized in that, The method further includes: The instance segmentation model is initially trained using base class data to obtain a pre-trained instance segmentation model; the base class data includes base class images, and the foreground category of the foreground object in the base class image is different from the foreground category of the foreground object in the sample image. The instance segmentation model trained using the sample data is the instance segmentation model after preliminary training.

8. An image processing apparatus, characterized in that, The device includes: The preprocessing module is used to perform feature extraction on the image to be processed and determine the target global feature map of the image to be processed. The preprocessing module is further configured to determine a target candidate region feature map based on the target global feature map, wherein the image region corresponding to the target candidate region feature map includes the foreground object in the image to be processed; The instance segmentation module is used to input the feature map of the target candidate region into the mask branch network in the instance segmentation model for processing, and to determine the foreground mask corresponding to the foreground object. The instance segmentation module is further configured to input the feature map of the target candidate region into the target detection branch network in the instance segmentation model for processing, and determine the foreground category and foreground location bounding box corresponding to the foreground object; The output module is used to determine the instance segmentation result of the image to be processed based on the foreground mask, foreground category, and foreground position bounding box corresponding to the foreground object; The instance segmentation model is trained using sample data, which includes sample images, synthesized images, a reference foreground mask for the sample images, a reference foreground category for the synthesized images, and reference foreground location bounding boxes. The synthesized images are obtained by combining foreground images extracted from the sample images and a portion of the background image. The foreground images include foreground objects in the sample images, and the foreground category of the foreground objects in the sample images is the target foreground category. The mask branch network is trained using an image segmentation loss, which is determined based on a predicted foreground mask and the reference foreground mask. The predicted foreground mask is obtained by processing the candidate region feature map of the sample images using the mask branch network. The object detection branch network is trained using an object classification loss and a bounding box regression loss, whereby the object classification loss is determined based on the predicted foreground category and the reference foreground category, and the bounding box regression loss is determined based on the predicted foreground location bounding boxes and the reference foreground location bounding boxes. The predicted foreground category and the predicted foreground location bounding boxes are obtained by processing the candidate region feature map of the synthesized images using the object detection branch network.

9. A computer device, characterized in that, include: The processor, the communication interface, and the storage device are interconnected, wherein the storage device stores executable program code, and the processor is used to call the executable program code to implement the image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that are executed by a processor to implement the image processing method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, are used to implement the image processing method as described in any one of claims 1-7.

Images