Image processing method, image processing device, computer device, and computer program

Abstract

The present disclosure provides an image processing method, apparatus, computer device, and storage medium applicable to an artificial intelligence scenario. The method includes the steps of: acquiring a set of candidate boxes by performing image detection of a critical region in a target image; acquiring a set of target boxes by performing a process for eliminating duplicates of the candidate boxes in the set of candidate boxes based on the target image; acquiring adjustment boxes by correcting the positions of the candidate boxes in the set of target boxes; and generating a set of adjustment boxes and a set of clipped images corresponding to the set of adjustment boxes based on the adjustment boxes, where the set of clipped images includes N clipped images; acquiring M segmented images associated with the critical region by performing a recognition and segmentation process on each of the N clipped images; and acquiring M correction boxes for indicating the critical region by performing a correction process on the adjustment boxes corresponding to the M segmented images in the set of adjustment boxes. According to the present disclosure, the accuracy of image detection can be improved.

Description

[Technical Field]

[0001] This application claims priority from a Chinese patent application filed with the China Patent Office on May 19, 2023, bearing application number 202310572375.4 and entitled "Image Processing Method, Apparatus, Computer Equipment, and Storage Medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer device, and storage medium. [Background technology]

[0003] Image detection is one of the key issues in computer vision research and provides an important foundation for understanding high-level semantic information in images. The difficulty of detecting critical regions in an image (e.g., an object's face) often stems from the diversity of the critical region itself, changes in viewing angle, lighting conditions, occlusion, and the influence of complex backgrounds. This means that image detection of critical regions in complex environments is a challenging task. Conventional detection methods can mark the specific location of the critical region in an image in the form of a bounding box. However, these methods only consider the single task of image detection and do not place enough emphasis on the accuracy of bounding box location, resulting in inaccurate bounding box location. That is, the detected bounding box may contain redundant information other than the critical region, or the information of the critical region may not be fully detected, significantly affecting the accuracy of image detection. Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, and storage medium that can improve the accuracy of image detection. [Means for solving the problem]

[0005] In one aspect of an embodiment of the present application, there is provided an image processing method, the method comprising: a step of acquiring a detection target image, and performing image detection of a critical portion of the detection target image to acquire a candidate box set, the candidate box set including a candidate box for marking an image detection result; a step of obtaining a set of target boxes by performing a process of eliminating duplicates of candidate boxes in the set of candidate boxes based on the detection target image, obtaining adjustment boxes by correcting the positions of the candidate boxes in the set of target boxes, and generating a set of adjustment boxes and a set of clipped images corresponding to the set of adjustment boxes based on the adjustment boxes, wherein the set of clipped images includes N clipped images (N is a positive integer), and one clipped image is obtained by clipping the detection target image based on one adjustment box in the set of adjustment boxes; performing a recognition and segmentation process on each of the N number of clipped images to obtain a segmentation result associated with a critical portion, the segmentation result including M number of segmented images (M is a positive integer equal to or less than N); The method includes a step of obtaining M correction boxes for indicating critical areas in the image to be detected by performing correction processing on the adjustment boxes corresponding to the M divided images in the set of adjustment boxes based on the coordinate positions of the M divided images.

[0006] In one aspect of an embodiment of the present application, there is provided an image processing device, the device comprising: an image detection module that, upon acquiring a detection target image, performs image detection of a critical portion of the detection target image to acquire a set of candidate boxes, the set of candidate boxes including candidate boxes for marking the image detection result; a deduplication adjustment module that performs deduplication processing on candidate boxes in a candidate box set based on a detection target image to obtain a processing target box set, corrects the positions of the candidate boxes in the processing target box set to obtain adjustment boxes, and generates an adjustment box set and a cutout image set corresponding to the adjustment box set based on the adjustment boxes, wherein the cutout image set includes N cutout images (N is a positive integer), and one cutout image is obtained by cutting out the detection target image based on one adjustment box in the adjustment box set; a recognition and segmentation module that performs a recognition and segmentation process on each of the N clipped images to obtain a segmentation result associated with a critical portion, the segmentation result including M (M is a positive integer equal to or less than N) segmented images; and a correction module that acquires M correction boxes for indicating critical areas in the detection target image by performing correction processing on the adjustment boxes corresponding to the M divided images based on the coordinate positions of the M divided images in the adjustment box set.

[0007] In one aspect of the present application, there is provided a computing device comprising a processor, a memory, and a network interface, The processor is connected to a memory and a network interface, the network interface providing a data communication function, the memory storing a computer program, and the processor calling the computer program to cause the computer device to execute the method provided in the embodiments of the present application.

[0008] In one aspect of an embodiment of the present application, a computer-readable storage medium is provided that stores a computer program, which, when loaded and executed by a processor, causes a computing device having the processor to perform a method provided in an embodiment of the present application.

[0009] In one aspect of an embodiment of the present application, a computer program product is provided that includes a computer program, the computer program being stored in a computer-readable storage medium, and when a processor of a computing device reads the computer program from the computer-readable storage medium and executes the computer program, the computing device is caused to perform a method in an embodiment of the present application. [Effects of the Invention]

[0010] In an embodiment of the present application, a computer device with image detection capabilities acquires a detection target image and performs image detection of critical regions in the detection target image to obtain a candidate box set, including candidate boxes for marking the image detection results. The candidate box set here is composed of bounding boxes in which critical regions initially detected by the computer device are located. To effectively ensure the accuracy of the bounding boxes in which the critical regions are located, the computer device can perform secondary adjustment processing (including deduplication processing, alignment correction, and correction processing) on ​​the candidate box set after acquiring the candidate box set. First, the computer device performs deduplication processing on the candidate boxes in the candidate box set based on the detection target image to obtain a target box set, and then performs alignment correction on the candidate boxes in the target box set to obtain adjustment boxes. Based on the adjustment boxes, the computer device can generate a set of adjustment boxes and a set of clipped images corresponding to the adjustment box set. The set of clipped images here includes N clipped images (N is a positive integer), and each clipped image is obtained by clipping the detection target image based on one adjustment box in the adjustment box set. Next, the computer equipment comprehensively considers the inherent relationship between the two tasks of image detection and image segmentation, and again performs a correction process on the candidate box set (i.e., the adjustment box set) that has undergone the de-duplication process and position correction process. In other words, by performing a recognition and segmentation process on each of the N cut-out images, it is possible to obtain a segmentation result associated with the critical portion. Furthermore, in the adjustment box set, it performs a correction process on the adjustment boxes corresponding to the M divided images included in the segmentation result based on the coordinate positions of the M divided images (M is a positive integer less than or equal to N), thereby accurately obtaining M bounding boxes (i.e., correction boxes) to indicate the critical portion in the image to be detected.As can be seen from this, the image detection method provided in the embodiments of the present application comprehensively takes into account the deduplication process, position correction, and the inherent relationship between the two tasks of image detection and image segmentation, and places importance on the index of the accuracy of determining the position of the bounding box in which the critical part is located, thereby obtaining a corrected box that can more accurately indicate the critical part of the image to be detected, and further effectively improving the accuracy of image detection.

[0011] In order to more clearly describe the configuration of the embodiments of the present application or the prior art, the following briefly introduces drawings necessary for describing the embodiments or the prior art. Obviously, the drawings in the following description only show some embodiments of the present application, and those skilled in the art can derive other drawings from these drawings without creative work. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a network architecture configuration provided in an embodiment of the present application. [Figure 2] 1 is a schematic diagram of a scenario for image detection of a critical portion of a detection target image provided in an embodiment of the present application; [Figure 3] FIG. 1 is a schematic diagram of the flow of an image processing method provided in an embodiment of the present application. [Figure 4] FIG. 1 is a schematic diagram of a scenario for performing image detection based on the image detection model provided in an embodiment of the present application. [Figure 5] FIG. 1 is a schematic diagram of a scenario for performing deduplication processing of a set of candidate boxes provided in an embodiment of the present application. [Figure 6] FIG. 1 is a schematic diagram of a scenario of position correction based on an image shift adjustment model provided in an embodiment of the present application. [Figure 7] FIG. 1 is a schematic diagram of a scenario in which recognition division processing is performed based on an image recognition division model provided in an embodiment of the present application. [Figure 8] FIG. 1 is a schematic diagram of a framework for image detection provided in an embodiment of the present application. [Figure 9]FIG. 1 is a schematic diagram illustrating the flow of a training method for an image detection model proposed in an embodiment of the present application. [Figure 10] FIG. 1 is a schematic diagram of the training method flow for the image shift adjustment model proposed in the embodiment of the present application. [Figure 11] FIG. 1 is a schematic diagram illustrating the flow of a training method for an image recognition segmentation model provided in an embodiment of the present application. [Figure 12] 1 is a schematic diagram illustrating the configuration of an image processing device provided in an embodiment of the present application. [Figure 13] FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, the configuration of the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. It is clear that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments that a person skilled in the art can obtain from the embodiments of the present application without creative work fall within the scope of protection of the present application.

[0014] It should be understood that an embodiment of the present application provides an iterative localization image detection method based on deep learning, which is applied to the field of artificial intelligence. Here, artificial intelligence (AI) refers to theories, methods, technologies, and application systems that use digital computers or computations controlled by digital computers to simulate, extend, or expand human intelligence, sense the environment, acquire knowledge, and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that aims to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of sensing, reasoning, and decision-making.

[0015] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware and software. Fundamental AI technologies generally include sensors, AI-specific chips, cloud computing, distributed memory, big data processing, operational / interactive systems, and mechatronics. AI software technologies primarily include computer vision technology, speech processing technology, natural language processing technology, and several aspects such as machine learning / deep learning, autonomous driving, and smart transportation.

[0016] Computer vision (CV) technology is the science that studies how machines "see"—more specifically, the use of cameras and computers to perform machine vision tasks such as target recognition, detection, and measurement, as well as graphics processing and computer processing to obtain images suitable for human observation or transmission to a device for detection. As a branch of science, CV studies related theories and technologies to build artificial intelligence systems that can extract information from images or multidimensional data. CV technology typically includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and mapping, autonomous driving, and smart transportation.

[0017] Machine learning (ML) is an interdisciplinary field that spans multiple disciplines, including probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory, and focuses on how computers imitate or realize human learning behavior, acquire new knowledge and skills, and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and a fundamental means of endowing computers with intelligence, and is applied across various fields of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and supervised learning.

[0018] Please refer to FIG. 1. FIG. 1 is a schematic diagram of a network architecture configuration provided in an embodiment of the present application. As shown in FIG. 1, this network architecture may include a server 10F and a terminal device cluster. The terminal device cluster may include one or more terminal devices. The number of terminal devices is not limited here. As shown in FIG. 1, the terminal device cluster may specifically include terminal devices 100a, 100b, 100c, ..., 100n. As shown in FIG. 1, the terminal devices 100a, 100b, 100c, ..., 100n can each be network-connected to the server 10F. This allows each terminal device to exchange data with the server 10F via this network connection. The network connection here is not limited to a specific connection method, and may be directly or indirectly connected via a wired communication method, directly or indirectly connected via a wireless communication method, or may be connected via another method. The present application is not limited thereto.

[0019] Here, each terminal device in the terminal device cluster may include a smart terminal with image processing capabilities, such as a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an in-vehicle terminal, or a smart TV. It should be understood that each terminal device in the terminal device cluster as shown in FIG. 1 may have a service application (i.e., an application client) installed. When executed on each terminal device, the application client can exchange data with the server 10F shown in FIG. 1 above. Here, the application client may include an application client such as a social client, a multimedia client (e.g., a video client), an entertainment client (e.g., a game client), an information stream client, an education client, or a live broadcast client. Here, the application client may be an independent client or an embedded sub-client embedded in a client (e.g., a social client, an education client, a multimedia client, etc.), and is not limited thereto.

[0020] 1, the server 10F in the embodiment of the present application may be a server corresponding to the application client. The server 10F may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services. Here, the embodiment of the present application does not limit the number of servers.

[0021] For ease of understanding, in the embodiment of the present application, one terminal device may be selected as a service terminal device from among the multiple terminal devices shown in Fig. 1. For example, in the embodiment of the present application, the terminal device 100a shown in Fig. 1 may be the service terminal device. A service application (i.e., an application client) may be embedded in this service terminal device. In this case, this service terminal device can exchange data with the server 10F via a service data platform corresponding to this application client.

[0022] In an embodiment of the present application, the computer device with image detection function may be a server or any one of the terminal devices in the terminal device cluster shown in Fig. 1, for example, the terminal device 100a. The specific type of the computer device is not limited here.

[0023] It should be understood that, when a computer device acquires a detection target image, it performs image detection of critical regions in the detection target image to acquire a candidate box set (the candidate box set includes candidate boxes for marking image detection results), and then performs a process to acquire a processing target box set by eliminating duplicates of the candidate boxes in the candidate box set based on the detection target image, and then performs position correction of the candidate boxes in the processing target box set to acquire adjustment boxes. Based on the adjustment boxes, the computer device needs to generate a set of adjustment boxes and a set of clipped images corresponding to the adjustment box set. Here, the set of clipped images includes N clipped images (N is a positive integer), and each clipped image is obtained by clipping the detection target image based on one adjustment box in the adjustment box set. Next, the computer device can obtain segmentation results associated with the critical regions by performing a recognition and segmentation process on each of the N clipped images, and further perform a correction process on the adjustment boxes corresponding to the M segmented images in the segmentation result based on the coordinate positions of the M segmented images in the adjustment box set. This allows M corrected boxes to accurately indicate the critical regions in the detection target image. Based on this, by comprehensively considering the relationship between the two tasks of image detection and image segmentation, image deduplication processing and position correction, and placing importance on the index of the accuracy of determining the position of the bounding box in which the critical part is located, a corrected box can be obtained that can more accurately indicate the critical part of the image to be detected, and further, the accuracy of image detection is effectively improved.

[0024] As can be understood, the image detection method provided in the embodiments of the present application can be widely applied to multiple application scenarios. The term "critical part" here refers to a part of interest of the detection target. If the critical part is a body part of a service object (e.g., a human, an animal, etc.), the image detection method can be applied to video production scenarios, social entertainment scenarios, identity recognition scenarios, security detection scenarios, human-computer interaction scenarios, medical health scenarios, etc. If the critical part is a violation object (e.g., a sensitive flag, a sensitive symbol), the image detection method can be applied to a violation object detection scenario. If the critical part is a detection object of interest (e.g., a brand logo for an advertisement), the image detection method can be applied to an implant detection scenario, etc.

[0025] For example, in video production scenarios, to generate enhanced functions such as cutting out character video clips, calculating character video appearance frequency, and displaying character-related information, a computer device often needs to extract critical frames in a video as target images for detection, and further needs to use such image detection methods in embodiments of the present application. Image detection and segmentation techniques can not only accurately detect the correction box where the critical portion (e.g., face) in the target image is located, but also obtain a segmented image corresponding to the correction box (e.g., a pixel-level face-foreground segmented image with the background removed). In this case, to realize the above-mentioned enhanced functions, the computer device can use the correction box and the segmented image corresponding to the correction box as preliminary data for subsequent character recognition.

[0026] In order to provide users with a richer social entertainment experience in social entertainment scenarios, a computer device may use an image acquired by the computer device as a detection target image in service applications such as social applications, game applications, and photo processing applications. Furthermore, such an image detection method is required in the embodiments of the present application. Image detection and segmentation technology can accurately detect a correction box in which a critical portion (e.g., a face) is located in the detection target image, and further realize functions such as selfie beautification, face transformation, and face animation at a higher level.

[0027] In an identity recognition scenario, when a computer device uses an image collected in real time as the target image, it must use the image detection method described in the present embodiment. Image detection and segmentation techniques are used to accurately detect a correction box in the target image where a critical feature (e.g., a face) is located. Information about the critical feature within the correction box is then recognized and compared with information in a database to achieve automatic recognition and authentication. For example, some service applications (e.g., financial applications) require face recognition using the image detection method. If face recognition is successful, the subsequent service (e.g., application login, asset transfer, etc.) can continue to be performed.

[0028] In a security detection scenario, cameras are installed in public places, commercial areas, schools, residential areas, etc. For example, the computer device may acquire an image collected by the camera and use it as a detection target image, and then detect a critical area, such as an animal's face, in the detection target image and perform subsequent image recognition based on the detected correction box. In order to find a lost pet more quickly, the lost pet's behavior path can be obtained in a timely manner by matching it with the pet information of the lost pet.

[0029] In a human-computer interaction scenario, the computer device can perform image detection of a face in the target image, and then, based on the detected correction box, can realize human-computer interaction functions, such as interaction functions for vending machines, smart homes, smart locks, etc.

[0030] In medical and health scenarios, in the medical field, image detection (e.g., face detection) can be performed on the target image of a patient, and subsequent recognition and monitoring can be performed based on the detected retrieval boxes to realize remote management. This image detection technology can also be applied to diagnosis and treatment. For example, diagnosing a patient's facial skin disease based on the face contained in the retrieval box.

[0031] Of course, the image detection method can be applied to other scenarios, but examples will not be given here. When the face (or other biometric feature) recognition technology according to the embodiments of the present application is applied to the specific products or technologies in the above embodiments of the present application, the collection, use, and processing of related data should comply with the requirements of national laws and regulations. Before collecting face information, information processing regulations should be notified, and the target object's independent consent should be sought. The face information should be processed in strict compliance with the requirements of laws and regulations and personal information processing regulations, and technical measures should be taken to ensure the security of related data.

[0032] For ease of understanding, please refer to FIG. 2. FIG. 2 is a schematic diagram of a scenario for image detection of a critical portion of a detection target image provided in an embodiment of the present application. As shown in FIG. 2, the computer device in the embodiment of the present application may be a computer device with an image detection function. The computer device may be any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, for example, the terminal device 100a. The computer device may be the server 10F shown in FIG. 1 above. The computer device is not limited here.

[0033] 2, the image 200p may be a detection target image acquired by the computer device. The image 200p may be any one of critical frame images in certain video data, a real-time image acquired by the computer device using an image acquisition component (e.g., a photo camera), or an image stored in the computer device (e.g., an image in an album). Examples will not be given here.

[0034] It should be understood that, once the computer device acquires the image 200p, the computer device may include an image detection step, an image deduplication and shift adjustment step, and an image recognition and segmentation step. In the image detection step, the computer device may first obtain a candidate box set by image detection of a critical portion (e.g., a face) in the image 200p. The candidate box set here refers to a set of bounding boxes for indicating the positions of the critical portions obtained by the computer device during the initial image detection of the image 200p. That is, the candidate box set includes candidate boxes for marking the image detection results. As shown in FIG. 2, the candidate box set may include seven candidate boxes, specifically candidate boxes 21B1, 21B2, 21B3, 21B4, 21B5, 21B6, and 21B7.

[0035] As can be seen, since the sliding window method is used for image detection, the candidate box set acquired by the computer device inevitably has the problem of some critical region overlapping. To effectively ensure the recall rate of image detection, the computer device needs to combine duplicate candidate boxes of the same critical region into one as much as possible, that is, perform a deduplication process. Based on this, in the image deduplication and shift adjustment step, the computer device obtains a target box set by deduplication of the candidate boxes in the candidate box set based on the image 200p, and obtains adjustment boxes by correcting the positions of the candidate boxes in the target box set. Based on the adjustment boxes, the computer device can generate an adjustment box set and a set of cropped images corresponding to the adjustment box set. Here, the set of cropped images may include N cropped images (N is a positive integer), and one cropped image is obtained by cropping the image 200p based on one adjustment box in the adjustment box set.

[0036] The deduplication shift adjustment process here refers to an image processing method including deduplication and position correction. The number of deduplication processes may be dynamically adjusted according to actual conditions. This is not a limitation. For example, the deduplication shift adjustment process may include one deduplication process and one position correction. That is, after performing the deduplication process on the set of candidate boxes, position correction of each candidate box in the set of candidate boxes that has been deduplication-processed is performed based on the image 200p. Optionally, the deduplication shift adjustment process may include two deduplication processes and one position correction. That is, after performing the deduplication process on the set of candidate boxes, position correction of each candidate box in the set of candidate boxes that has been deduplication-processed is performed based on the image 200p. Then, deduplication is performed again on the adjusted boxes after position correction.

[0037] 2, the adjustment box set acquired by the computer device may include three adjustment boxes, specifically adjustment boxes 22B1, 22B2, and 22B3. Based on this, the cropped image set corresponding to the adjustment box set may include three cropped images, specifically cropped image 1, cropped image 2, and cropped image 3. Here, cropped image 1 is the cropped image obtained by cropping image 200p based on adjustment box 22B1, cropped image 2 is the cropped image obtained by cropping image 200p based on adjustment box 22B2, and cropped image 3 is the cropped image obtained by cropping image 200p based on adjustment box 22B3.

[0038] Furthermore, in the image recognition and segmentation stage, the computer device needs to comprehensively consider the inherent relationship between the two tasks of image detection and image segmentation, and re-perform a correction process on the candidate box set (i.e., the adjusted box set) after the de-duplication and shift adjustment process according to the segmentation result of the recognition and segmentation process. The segmentation result here may include M (M is a positive integer less than or equal to N) segmented images. The segmented images refer to images obtained by segmenting a cutout image belonging to the recognized critical region (for example, a pixel-level face foreground segmented image with the background removed).

[0039] As shown in FIG. 2, the computer device can perform a recognition and segmentation process on each of the three clipped images. This allows a segmentation result associated with the critical region to be obtained. Specifically, the segmentation result may include a segmented image P1 and a segmented image P2. In this case, the computer device needs to perform a correction process on the adjustment boxes corresponding to the two segmented images in the adjustment box set. For example, the computer device needs to perform a correction process on the adjustment box 22B1 in the adjustment box set based on the coordinate position of the segmented image P1 to obtain a correction box 23B1. At the same time, the computer device needs to perform a correction process on the adjustment box 22B3 in the adjustment box set based on the coordinate position of the segmented image P2 to obtain a correction box 23B2. This means that after the computer device in this embodiment performs the correction process on the adjustment box set based on the segmented images in the segmentation result, two bounding boxes (i.e., correction boxes) can be obtained to indicate the critical region in the image 200p. This means that the faces of two different service objects exist in the image 200p.

[0040] As can be seen, in the embodiment of the present application, when detecting the critical regions of image 200p, the image detection stage, the image deduplication and shift adjustment stage, and the image recognition and segmentation stage are linked together to complete an end-to-end closed-loop design for image detection and image segmentation, and further achieve the goal of focusing on the accuracy of determining the location of the bounding box where the critical region is located. That is, the embodiment of the present application can satisfy the rapid detection of complex critical regions (e.g., faces) in the target image, and has the characteristics of high precision, high recall, and extremely high accuracy in determining the location of the bounding box. It also has multi-functional characteristics, such as being able to output a modified box and being able to output a segmented image after pixel-level background removal.

[0041] Here, when a computer device with an image detection function detects the critical parts of an image to be detected, the specific implementation method for obtaining a correction box to indicate the critical parts in the image to be detected by comprehensively considering the deduplication / shift adjustment process and the inherent relationship between the two tasks of image detection and image segmentation can be seen in the examples corresponding to Figures 3 to 11 below.

[0042] Further, please refer to FIG. 3. FIG. 3 is a schematic diagram of the flow of an image processing method provided in an embodiment of the present application. As shown in FIG. 3, the method may be executed by a computer device having an image detection function. The computer device may be a terminal device (e.g., any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, e.g., terminal device 100a) or a server (e.g., server 10F shown in FIG. 1 above). This is not limiting. For ease of understanding, the embodiment of the present application will be described taking the case where the method is executed by a server as an example. The method may include at least the following steps S101 to S104.

[0043] In step S101, when a detection target image is acquired, a set of candidate boxes is acquired by image-detecting critical portions of the detection target image.

[0044] The candidate box set here is a set of critical regions in a target image detected by a computer system using an image detection model. The candidate box set includes candidate boxes for marking the image detection results (the image detection results may indicate possible regions detected in the image, and the candidate boxes are used to mark these possible regions). For example, the candidate box set may include K candidate boxes (K is a positive integer). The image detection model is further used to determine a predicted classification confidence for each candidate box in the candidate box set. The predicted classification confidence here refers to the probability that the classification type corresponding to the candidate box belongs to the critical region category. A higher predicted classification confidence indicates a higher likelihood that the candidate box contains a critical region. The model input of the image detection model is the target image, and the model output of the image detection model is whether a critical region exists in the target image. If a critical region exists in the target image, the coordinate position of the candidate box where the critical region is located is output in the format [x, y, w, h]. Here, x and y refer to the coordinates of a vertex of the candidate box (e.g., the top left coordinates), w refers to the width corresponding to the candidate box, and h refers to the height corresponding to the candidate box.

[0045] As can be understood, the image detection model may automatically crop the target image based on the coordinate position of the candidate box, thereby outputting a specific cropped image containing the critical region. In other words, the image detection model in this embodiment is primarily responsible for detecting the target image with a very high coverage rate. This determines whether the target image contains a pseudo-critical region. If the target image contains a pseudo-critical region, the bounding box (i.e., the candidate box) containing the critical region can be cropped and stored in a picture format (e.g., jpg format), and the image file can be obtained.

[0046] Here, the image detection model is the first step in the image detection process in the embodiment of the present application, and its computation speed largely determines the computation speed of the entire process. However, at the same time, the expressive power of the image detection model must be strong enough to complete the coarse detection task. Based on the above two factors, the image detection model in the embodiment of the present application may include two network structures: a first network structure in which the model input accepts a fixed image size, and a second network structure in which the model input accepts any image size. Compared to the first network structure, the second network structure is a fully convolutional network structure, i.e., all feature processing layers in the network are convolutional layers.

[0047] Based on this, if the network structure of the image detection model called by the computer device is the first network structure, the computer device must perform a scaling process on the detection target image. This scales the image size of the detection target image to a fixed image size (e.g., 48*48*3) that is acceptable for reception by the first network structure. The scaled detection target image is then directly input to the image detection model, and the image detection model can then perform image detection of critical portions of the scaled detection target image. The multiple candidate boxes that have been image-detected are then considered to be a candidate box set.

[0048] When the network structure of the image detection model called by the computer device is the second network structure, the computer device does not need to spend much time, but directly inputs the detection target image into the image detection model, and uses the image detection model to image-detect the critical parts of the detection target image. Furthermore, the multiple image-detected candidate boxes are defined as a candidate box set.

[0049] For ease of understanding, please refer to FIG. 4. FIG. 4 is a schematic diagram of a scenario in which image detection is performed based on the image detection model provided in the embodiment of the present application. As shown in FIG. 4, an image 400p is a detection target image to be recognized, acquired by a computer device in the embodiment of the present application. The computer device may be any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, for example, the terminal device 100a. The computer device may also be the server 10F shown in FIG. 1 above. The computer device is not limited here.

[0050] It should be understood that when the computer device acquires an image 400p, it can call the image detection model 40W shown in FIG. 4. Here, the image detection model 40W may include a convolutional layer 4L1, a max pooling layer 4L2, a normalization layer 4L3, a convolutional layer 4L4, a normalization layer 4L5, a max pooling layer 4L6, a network layer 4L7, and an output layer 4L8. For example, the convolutional layer 4L1 and the convolutional layer 4L4 may use 64 5*5 convolution kernels with a step size of 1. The filter parameters of the max pooling layer 4L2 and the max pooling layer 4L6 may both be 3*3 with a step size of 2. The output layer 4L8 may be a two-output node for outputting a classification, such as a critical area or a non-critical area.

[0051] Here, if the network layer 4L7 belongs to a fully connected layer, the network structure of the image detection model 40W belongs to the first network structure, which means that a detection target image of a fixed image size needs to be input. Based on this, when the computer device acquires the image 400p, it needs to perform a scaling process on the image 400p. This scales the image 400p to a fixed image size (e.g., 48*48*3) that can be received by the first network structure. Furthermore, the scaled image 400p can be directly input to the image detection model 40W, and the image detection model 40W can perform image detection of critical portions of the scaled image 400p. Furthermore, the multiple candidate boxes output from the image detection model 40W can be defined as a candidate box set. Here, the candidate box set may specifically include candidate box 40B1, candidate box 40B2, and candidate box 40B3 shown in FIG. 4.

[0052] Optionally, if the network layer 4L7 belongs to a convolutional layer, it means that the network structure of the image detection model 40W is a fully convolutional network structure. Therefore, the computer device can determine that the network structure of the image detection model 40W belongs to a second network structure. This means that the image detection model 40W can complete two tasks of image detection, namely, bounding box detection (detection proposal) and classification (classification), by sliding detection of an image of any size in a sliding window with a fixed step size.

[0053] Based on this, the computer device can directly input the image 400p into the image detection model 40W, and use the image detection model 40W to image-detect the critical region in the image 400p. Furthermore, a plurality of candidate boxes output from the image detection model 40W can be defined as a candidate box set. Here, the candidate box set may specifically include candidate box 40B1, candidate box 40B2, and candidate box 40B3 shown in FIG. 4. Because the image detection model 40W can perform sliding-window detection of images of any size and detect as many critical regions as possible from the image with an extremely high coverage rate, the image detection model can meet the characteristics of the task and detect critical regions of any size in images of any size in a very short time, providing good pre-screening for subsequent processes, improving the execution efficiency of the entire system, and avoiding a large amount of unnecessary calculations.

[0054] In step S102, a set of target boxes is obtained by performing a process to eliminate duplicates of the candidate boxes in the set of candidate boxes based on the image to be detected, and an adjustment box is obtained by correcting the position of the candidate boxes in the set of target boxes, and based on the adjustment box, an adjustment box set and a set of cut-out images corresponding to the adjustment box set are generated.

[0055] Specifically, the computer device may obtain a set of target boxes by performing a process of deduplication on the set of candidate boxes based on the shape adaptive maximum suppression deduplication rule. i where i is a positive integer equal to or less than H, and H represents the total number of candidate boxes in the set of target boxes. Furthermore, the computer device invokes an image shift adjustment model to calculate the candidate boxes X according to the target image. i By correcting the position of the adjustment box Y i and adjustment box Y iand clipped images corresponding to the H adjustment boxes. After obtaining the H adjustment boxes, the computer device subsequently performs a deduplication process on the H adjustment boxes based on the deduplication rules, determines the adjustment boxes after the deduplication process as an adjustment box set corresponding to the candidate box set, and determines the clipped images corresponding to the adjustment boxes after the deduplication process as a clipped image set corresponding to the adjustment box set. Here, the clipped image set includes N clipped images (N is a positive integer), and one clipped image is obtained by clipping the detection target image based on one adjustment box in the adjustment box set.

[0056] As can be seen, an image detection model can effectively detect pseudo-critical regions (e.g., pseudo-faces) in a target image. However, due to limitations in the network's performance and the amount of training data, the image detection model may mistakenly detect some regions that are similar to critical regions as critical regions. Furthermore, the sliding window image detection method inevitably leads to the problem of overlapping face regions. Therefore, the computer device must further perform a deduplication process for the set of candidate boxes based on a shape-adaptive maximum suppression deduplication rule. Here, the deduplication rule mainly performs deduplication of candidate boxes in which pseudo-critical regions are located within the set of candidate boxes, and consolidates duplicate candidate boxes of the same object into one as much as possible while maintaining recall.

[0057] Non-Maximum Suppression (NMS) refers to extracting the most representative data from multiple overlapping data and suppressing other overlapping data. As the name suggests, it suppresses non-maximal elements, and can be understood as a local maximum search. NMS uses certain conditions to select locally overlapping candidate boxes and obtain the optimal candidate box. While the NMS algorithm is widely applied in object detection algorithms, considering the shape characteristics of critical regions (e.g., faces), faces are not squares with the same length and width, but are approximate ellipses with their length slightly greater than their width. This means that simply applying a conventional NMS algorithm to remove bounding boxes containing overlapping critical regions may remove some face boxes that are close to each other but are actually two different faces. This is because many non-face regions exist on both the left and right sides of a rectangular face box, causing the NMS algorithm to misjudge the overlap between the two face boxes. Therefore, in the embodiment of the present application, a shape-adaptive non-maximum suppression algorithm can be obtained by improving the normal non-maximum suppression algorithm, and further, the duplicate elimination process can be performed based on this algorithm.

[0058] If the candidate box set includes K candidate box sets, the computer device may obtain K clipping boxes by clipping each of the K candidate boxes based on an image clipping rate set by a shape adaptive maximum suppression deduplication rule. Furthermore, since the image detection model is also used to determine a predicted classification confidence corresponding to each of the K candidate boxes, the computer device may obtain a sorted result by sorting the K clipping boxes based on the K predicted classification confidences. Here, the computer device may determine the clipping box with the highest predicted classification confidence in the sorted result as a first clipping box, and determine the (K-1) clipping boxes other than the first clipping box in the sorted result as a set to be filtered.

[0059] Next, the computer device may obtain a set of reserved boxes by performing a deduplication process on the K clipping boxes based on the overlap between the first clipping box and each clipping box in the filtering target set. Here, as can be understood, the computer device may determine the overlap between the first clipping box and each clipping box in the filtering target set. If an overlapping clipping box whose overlap is greater than an overlap threshold exists in the filtering target set, the computer device may retain the first clipping box and filter the overlapping clipping box in the filtering target set. Furthermore, in the filtering target set after filtering, the clipping box with the highest predicted classification reliability may be determined as the second clipping box, and the clipping boxes other than the second clipping box may be determined as a new filtering target set. Next, the computer device may retain the second clipping box and perform a deduplication process on the new filtering target set based on the overlap between the second clipping box and each clipping box in the new filtering target set until the filtering target set after the deduplication process is empty, and determine the reserved clipping box as the set of reserved boxes. The set of reserved boxes includes a first clipping box and a second clipping box.

[0060] Specifically, the specific embodiment of the computer device performing the clipping process of the candidate box set based on the shape adaptive maximum suppression duplicate elimination rule can be referred to as Equation 1 below.

number

[0061] Here, (x1, y1) is the first vertex coordinate (e.g., the top left coordinate) of a candidate box in the candidate box set, (x2, y2) is the second vertex coordinate (e.g., the bottom right coordinate) that is diagonally related to the first vertex coordinate in the candidate box, and narrowrate is the image cropping rate set in the duplicate elimination rule, e.g., 0.08. This image cropping rate is the optimal value obtained through multiple experiments.

[0062] Specifically, the specific embodiment in which the computer device determines the overlap between any two bounding boxes (for example, bounding box 1 and bounding box 2) can be seen from Equation 2 below.

number

[0063] where C 12 refers to the overlap area between bounding box 1 and bounding box 2, and U 12 refers to the union area between Bounding Box 1 and Bounding Box 2.

[0064] Finally, the computer device needs to obtain a set of target boxes by performing image expansion processing on each of the clipping boxes in the set of reserved boxes according to the image cropping rate. Specifically, the specific embodiment of the computer device performing image expansion processing on the clipping boxes in the set of reserved boxes can be seen from Equation 3 below:

number

[0065] Here, (x1, y1) is the first vertex coordinate (e.g., the top left coordinate) of a clipping box in the set of reserved boxes, (x2, y2) is the second vertex coordinate (e.g., the bottom right coordinate) that is diagonally related to the first vertex coordinate in the clipping box, and narrowrate may be the image clipping rate set in the duplicate elimination rules, for example, 0.08.

[0066] For ease of understanding, further reference is made to FIG. 5. FIG. 5 is a schematic diagram of a scenario for performing a duplicate elimination process on a set of candidate boxes provided in an embodiment of the present application. As shown in FIG. 5, an image 500p is a detection target image awaiting recognition, acquired by a computer device in an embodiment of the present application. The computer device may be any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, for example, the terminal device 100a. The computer device may also be the server 10F shown in FIG. 1 above. The computer device is not limited here.

[0067] 5, the set of candidate boxes acquired by the computer device after performing step S101 may include five candidate boxes, specifically, candidate box 50B1, candidate box 50B2, candidate box 50B3, candidate box 50B4, and candidate box 50B5. The image detection model may also output predicted classification confidences corresponding to each of the five candidate boxes (specifically, predicted classification confidence 1 (e.g., 0.9) corresponding to candidate box 50B1, predicted classification confidence 2 (e.g., 0.85) corresponding to candidate box 50B2, predicted classification confidence 3 (e.g., 0.82) corresponding to candidate box 50B3, predicted classification confidence 4 (e.g., 0.87) corresponding to candidate box 50B4, and predicted classification confidence 5 (e.g., 0.81) corresponding to candidate box 50B5).

[0068] In this case, the computer device may perform a clipping process (e.g., a clipping operation on the left and right sides) on each of the five candidate boxes based on the above formula 1 and the image clipping rate set in the duplicate elimination rule. This allows a set of clipping boxes to be obtained. For example, the computer device may perform a clipping process on candidate box 50B1 based on the above formula 1 and the image clipping rate to obtain clipping box 51B1 corresponding to candidate box 50B1. By this analogy, the computer device may obtain clipping box 51B2 corresponding to candidate box 50B2, clipping box 51B3 corresponding to candidate box 50B3, clipping box 51B4 corresponding to candidate box 50B4, and clipping box 51B5 corresponding to candidate box 50B5.

[0069] Furthermore, the computer device may perform a sorting process on the five clipping boxes in the clipping box set based on the five predicted classification confidences. Thus, a sorting result can be obtained. For example, the sorting result may be clipping box 51B1, clipping box 51B4, clipping box 51B2, clipping box 51B3, and clipping box 51B5 shown in FIG. 5 . Based on this, the computer device may determine the clipping box with the highest predicted classification confidence in the sorting result as a first clipping box (e.g., clipping box 51B1), and determine the four clipping boxes other than the first clipping box in the sorting result as a filtering target set (e.g., clipping box 51B4, clipping box 51B2, clipping box 51B3, and clipping box 51B5).

[0070] Based on the above formula 2, the computer device can determine the overlap between the clipping box 51B1 and each clipping box in the filtering target set (specifically, this may include the overlap between the clipping box 51B1 and the clipping box 51B4 (e.g., 0), the overlap between the clipping box 51B1 and the clipping box 51B2 (e.g., 0.7), the overlap between the clipping box 51B1 and the clipping box 51B3 (e.g., 0.5), and the overlap between the clipping box 51B1 and the clipping box 51B5 (e.g., 0)).

[0071] Then, the computer device can determine whether there is an overlapping clipping box in the filtering target set based on the overlapping threshold. Here, the overlapping clipping box refers to a clipping box whose overlapping degree with the first clipping box reaches an overlapping threshold (e.g., 0.4). The overlapping threshold may be dynamically adjusted according to specific circumstances, but is not limited thereto.

[0072] Since the overlap between clipping box 51B1 and clipping box 51B2 and the overlap between clipping box 51B1 and clipping box 51B3 both reach the overlap threshold, the computer device can determine clipping box 51B2 and clipping box 51B3 as overlapping clipping boxes, and further, by retaining clipping box 51B1 and filtering the overlapping clipping box in the filtering target set, the filtered filtering target set (for example, clipping box 51B4 and clipping box 51B5) can be obtained.

[0073] Because the filtered filtering target set is not empty, the computer device needs to continue scanning the filtered filtering target set to determine whether there is an overlapping clipping box in the filtered filtering target set. That is, the computer device may select the clipping box with the highest predicted classification confidence in the filtered filtering target set as a second clipping box (e.g., clipping box 51B4), and select a clipping box other than the second clipping box as a new filtering target set (e.g., clipping box 51B5).

[0074] Next, because one clipping box exists in the new filtering target set, the computer device can determine the overlapping degree (e.g., 0.8) between clipping box 51B4 and clipping box 51B5 based on the above formula 2. Because the overlapping degree between clipping box 51B4 and clipping box 51B5 reaches the overlapping degree threshold, the computer device can determine clipping box 51B5 as an overlapping clipping box, and further reserve clipping box 51B4 and perform a deduplication process on the overlapping clipping box in the new filtering target set. Because the filtering target set is empty after the deduplication process, the computer device can determine that scanning of all clipping box sets has been completed. At this time, the computer device can determine the reserved first clipping box (i.e., clipping box 51B1) and the reserved second clipping box (i.e., clipping box 51B4) as a reserved box set.

[0075] After completing the filtering of the overlapping clipping boxes, the computer device may expand each clipping box in the set of reserved boxes again into a square bounding box. For example, the computer device may obtain the candidate box 52B1 shown in FIG. 5 by performing an image expansion process on the clipping box 51B1 based on the above Equation 3 and the image clipping rate. Similarly, the computer device may obtain the candidate box 52B4 shown in FIG. 5 by performing an image expansion process on the clipping box 51B4 based on the above Equation 3 and the image clipping rate. In this case, the computer device may determine the candidate box 52B1 and the candidate box 52B4 as a set of boxes to be processed.

[0076] As can be seen, in the embodiment of the present application, by replacing the conventional NMS algorithm with a shape-adaptive non-maximum suppression algorithm, the phenomenon of NMS misoperation can be avoided to a certain extent. Based on this shape-adaptive non-maximum suppression deduplication rule, duplicate candidate boxes in a candidate box set can be deduped and a few most representative candidate boxes can be reserved. Because the deduplication rule is a shape-adaptive optimization that takes into account the special shape of a critical part (e.g., a face), this optimization is more in line with service characteristics, effectively deduping face boxes belonging to the same object and retaining face boxes of different objects that are too close, thereby significantly reducing the number of face boxes that need to be processed in subsequent steps and ensuring detection coverage.

[0077] Furthermore, the computer device needs to invoke an image shift adjustment model to perform a shift prediction for each candidate box in the set of target boxes and iteratively correct the position of each candidate box. The model-based structure of the image shift adjustment model here may be based on a convolutional neural network, an attention model, or the like. The model-based structure of the image shift adjustment model is not limited here.

[0078] The model input of the image shift adjustment model is the coordinate position of the area where each candidate box in the set of boxes to be processed is located (i.e., the coordinate position of the candidate box), and the detection target image, and the model output of the image shift adjustment model is the candidate box whose position has been corrected (i.e., the adjustment box). As can be seen, the image shift adjustment model can output the coordinate of the specific area where the adjustment box is located, and the coordinate position of the adjustment box is in the format of [x, y, w, h], where x and y refer to the coordinates of a vertex of the adjustment box (e.g., the upper left coordinate), w refers to the width corresponding to the adjustment box, and h refers to the height corresponding to the adjustment box.

[0079] For example, the computer device can call an image shift adjustment model, and further calculate a candidate box X according to the target image. i Then, the computer device calculates the candidate box X based on the first regression parameters. i By correcting the position of the candidate box X i A first adjustment box corresponding to the first regression parameter can be obtained, and whether the adjustment is completed can be determined based on the relationship between the first regression parameter and the regression parameter threshold range. The regression parameter threshold range can be dynamically adjusted according to actual needs. This is not a limitation. For example, the regression parameter threshold range can be [-2, 2].

[0080] If the first regression parameter falls within the regression parameter threshold range, the computer device can determine that the adjustment is completed, and further select the candidate box X i The first adjustment box corresponding to the candidate box X i Corresponding adjustment box Y iOptionally, if the first regression parameter does not belong to the regression parameter threshold range, the computer device may determine that the position of the first adjustment box needs to be further corrected, that is, invoke the image shift adjustment model, and determine the candidate box X according to the detection target image. i By predicting the deviation of the first adjustment box corresponding to X, the second regression parameters can be obtained. Then, the candidate box X can be calculated based on the second regression parameters. i By adjusting the first adjustment box corresponding to i If the second regression parameter falls within the regression parameter threshold range, the second adjustment box corresponding to the candidate box X i The second adjustment box corresponding to the candidate box X i Corresponding adjustment box Y i and then adjust the Y i The adjustment box Y is created by cutting out the detection target image based on the coordinate position of i A corresponding cutout image can be obtained.

[0081] Specifically, for a specific embodiment in which the computer device performs the adjustment process for a certain candidate box based on the regression parameter (for example, bbr), see Equations 4 and 5 below.

number

number

[0082] Here, x0 and y0 indicate the coordinates of a vertex of the current candidate box (for example, the upper left coordinate), w0 indicates the width corresponding to the current candidate box, and h0 indicates the height corresponding to the candidate box. bbr is a regression parameter obtained by performing a shift prediction process on the coordinate position of the current candidate box using an image shift adjustment model. Here, x offsetrefers to the predicted deviation obtained for the coordinate (e.g., the abscissa) corresponding to x0, and y offset refers to the predicted deviation obtained for the coordinate (e.g., the ordinate) corresponding to y0, and w offset refers to the predicted deviation obtained for the width, and h offset refers to the predicted deviation obtained for height.

[0083] For easier understanding, please refer to Fig. 6. Fig. 6 is a schematic diagram of a scenario of performing position correction based on the image shift adjustment model provided in the embodiment of the present application. As shown in Fig. 6, an image 600p is a detection target image to be recognized, which is acquired by a computer device in the embodiment of the present application. The candidate box 60B1 and the candidate box 60B2 in the processing target box set are both acquired after the computer device performs a duplicate elimination process on the candidate box set.

[0084] As shown in Figure 6, the computer device may call the image shift adjustment model 60W after performing a duplicate elimination process on the candidate box set. For example, the network structure of the image shift adjustment model 60W is different from that of the image detection model. The shift mode classification task is more difficult than the critical region classification task. Therefore, multiple convolution operations are added to the network to extract more expressive features. The image shift adjustment model 60W can receive 48*48*3 data input and output 45 types of shift mode classification results.

[0085] As shown in FIG. 6, the image shift adjustment model 60W may specifically include a convolutional layer 6L1, a max pooling layer 6L2, a convolutional layer 6L3, a max pooling layer 6L4, a convolutional layer 6L5, a max pooling layer 6L6, a convolutional layer 6L7, a fully connected layer 6L8, and an output layer 6L9. For example, the convolutional layer 6L1 may use 32 3*3 convolutional kernels with a step size of 1; the filter parameters of the max pooling layer 6L2 may be 3*3 with a step size of 2; the convolutional layer 6L3 may use 64 3*3 convolutional kernels with a step size of 1; the filter parameters of the max pooling layer 6L4 may be 3*3 with a step size of 1; the convolutional layer 6L5 may use 64 3*3 convolutional kernels with a step size of 1; the filter parameters of the max pooling layer 6L6 may be 2*2 with a step size of 2; the convolutional layer 6L7 may use 128 2*2 convolutional kernels with a step size of 1; the output size of the fully connected layer 6L8 may be 1*256; and the output layer 6L9 may have 45 output nodes for outputting 45 shift mode classes.

[0086] It should be understood that the computer device may directly input the two candidate boxes in the set of boxes to be processed and the image 600p into the image shift adjustment model 60W, thereby performing shift prediction for the two candidate boxes respectively, and obtaining corresponding predicted regression parameters.

[0087] For ease of understanding, please refer to Table 1. Table 1 is a shift adjustment table determined based on the image shift adjustment model provided in the embodiment of the present application. Here, in the shift adjustment table shown in Table 1, each candidate box may include two-dimensional results. One is the predicted classification confidence of the candidate box determined by the image shift adjustment model, and the other is the regression parameter of the candidate box determined by the image shift adjustment model. Specifically, as shown in Table 1,

[0088] [Table 1]

[0089] For example, for candidate box 60B1, the coordinate position of the candidate box 60B1 is [x1, y1, w1, h1], and the predicted regression parameter of the candidate box 60B1 is bbr1. Furthermore, the computer device may perform position correction of the coordinate position of the candidate box 60B1 based on the above Equation 5 and the regression parameter bbr1 (i.e., the first regression parameter), and may further determine the candidate box 60B1 after position correction as a first adjusted box corresponding to the candidate box 60B1.

[0090] As can be seen, due to the complexity of the bounding box in which the critical region is located, a single fine-tuning of the position may not achieve the optimal effect. Therefore, the computer device needs to determine whether the first adjustment box needs to be re-input into the image shift adjustment model 60W shown in Figure 6 for the next fine-tuning (i.e., position correction) based on the first regression parameters. That is, in the embodiment of the present application, by repeatedly and iteratively correcting the position of the candidate box, more accurate position determination can be achieved, and the finally obtained adjustment box has extremely high position determination accuracy, especially, it can accurately determine the position of extremely small critical regions and effectively distinguish adjacent or locally overlapping critical regions.

[0091] As can be seen from Table 1 above, the regression parameter bbr1 does not fall within the regression parameter threshold. Therefore, the computer device needs to re-input the first adjustment box corresponding to the candidate box 60B1 into the image shift adjustment model 60W shown in FIG. 6. Then, using the image 600p, shift prediction is performed again on the first adjustment box corresponding to the candidate box 60B1 to obtain new predicted regression parameters (i.e., second regression parameters). Furthermore, based on the second regression parameters, an adjustment process for the first adjustment box corresponding to the candidate box 60B1 can be performed, and a second adjustment box corresponding to the candidate box 60B1 can be obtained. If the new predicted regression parameters fall within the regression parameter range, it is deemed that fine adjustment (position correction) is no longer necessary. At this time, the finally output adjustment box (i.e., second adjustment box) can be determined as the adjustment box corresponding to the candidate box 60B1 (e.g., adjustment box 61B1 shown in FIG. 6). Furthermore, the computer device can obtain a clipped image corresponding to the adjustment box 61B1 by clipping the image 600p based on the adjustment box 61B1.

[0092] Similarly, in the examples of the present application, by referring to a specific embodiment of performing the shift adjustment processing of the candidate box 60B1, the shift adjustment processing of the candidate box 60B2 can be performed, so that the adjustment box corresponding to the candidate box 60B2 (for example, the adjustment box 61B2 shown in FIG. 6) and the cut-out image corresponding to the adjustment box 61B2 can be obtained.

[0093] As can be seen, in order to effectively ensure the accuracy of the bounding boxes where the critical regions are located, the parameters used when performing the deduplication process (i.e., the initial deduplication) of the candidate box set are often loose (e.g., the overlap threshold is set large). That is, among the H candidate boxes in the target box set after the initial deduplication, there is still a possibility that two candidate boxes that overlap to some extent exist. Furthermore, when performing the shift adjustment process using the image shift adjustment model, the candidate boxes where the critical regions of two different objects are located are adjusted farther apart, and the candidate boxes where the critical regions of two same objects are located are adjusted closer together. This means that there is still a possibility that overlapping adjusted boxes exist in the target box set after adjustment. In order to effectively improve the efficiency of subsequent detection, the computer device needs to perform the deduplication process again after obtaining the H adjusted boxes. Furthermore, the adjusted boxes after the deduplication process may be determined as the adjusted box set corresponding to the candidate box set as input for the next model (e.g., an image recognition segmentation model), and the cropped images corresponding to the adjusted boxes after the deduplication process may be determined as the cropped image set corresponding to the adjusted box set.

[0094] In step S103, a recognition and division process is performed on each of the N cutout images to obtain a division result associated with the critical portion.

[0095] Specifically, the computer device may determine a target image from among the N clipped images, call an image recognition segmentation model, recognize and process the target image using an image classification branch in the image recognition segmentation model to obtain a predicted classification reliability corresponding to the target image, and then remove and process non-critical portions of the target image using the image segmentation branch in the image recognition segmentation model to obtain segmented images corresponding to the target image. If the predicted classification reliability corresponding to the target image is greater than a reliability threshold, the target image is deemed to belong to a classification category called critical portion. In this case, the computer device may determine the segmented images corresponding to the target image as segmentation results associated with the critical portion. Here, the segmentation results include M segmented images (M is a positive integer equal to or less than N).

[0096] For ease of understanding, further reference is made to FIG. 7. FIG. 7 is a schematic diagram of a scenario in which recognition segmentation processing is performed based on the image recognition segmentation model provided in the embodiment of the present application. As shown in FIG. 7, the set of cropped images may include N cropped images (N is a positive integer). One cropped image is obtained by a computer device cropping a detection target image based on one adjustment box in the adjustment box set. For convenience of explanation, the case where N is three may be taken as an example, and specifically, cropped image 70P1, cropped image 70P2, and cropped image 70P3 may be included. The detection target image here may be image 200p shown in FIG. 2 above. Therefore, cropped image 70P1 is obtained by a computer device cropping out image 200P based on adjustment box 22B1 shown in Figure 2, cropped image 70P2 is obtained by a computer device cropping out image 200P based on adjustment box 22B2 shown in Figure 2, and cropped image 70P3 is obtained by a computer device cropping out image 200P based on adjustment box 22B3 shown in Figure 2.

[0097] To learn more detailed information and distinguish between the bounding boxes that were not correctly detected in the previous two steps, the image recognition segmentation model used in this embodiment is composed of a public network layer and a multi-task branch. The public network layer may include multiple convolutional layers and multiple residual bottleneck structures, and the pooling layers in the image recognition segmentation model may all be average pooling layers.

[0098] As shown in FIG. 7, the image recognition segmentation model 70W may include a public network layer 70G, an image classification branch 71f, and an image segmentation branch 72f. The public network layer 70G may include a convolutional layer 70L1, a convolutional layer 70L2, a convolutional layer 70L3, a pooling layer 70L4, and multiple bottleneck structures 70L5. For example, the convolutional layer 70L1 may use 64 3*3 convolutional kernels with a step size of 2; the convolutional layer 70L2 may use 64 3*3 convolutional kernels with a step size of 1; the convolutional layer 70L3 may use 128 3*3 convolutional kernels with a step size of 1; and the pooling layer 70L4 may have filter parameters of 3*3 and a step size of 2. The multiple bottleneck structures may include four examples, as shown in Table 2 below. Table 2 is a schematic table of the parameters of the bottleneck structure group provided in the embodiment of the present application. As shown in Table 2,

[0099] [Table 2]

[0100] Here, the image classification branch may include a convolution layer 71L1, a convolution layer 71L2, a pooling layer 71L3, a convolution layer 71L4, and an output node layer 71L5. For example, the convolution layer 71L1 and the convolution layer 71L2 may each use 512 3*3 convolution kernels with a step size of 2. The pooling layer 71L3 may have filter parameters of 7*7 with a step size of 7. The convolution layer 71L4 may use two 1*1 convolution kernels with a step size of 1. The output node layer 71L5 may be a two-output node for outputting a classification category, i.e., a critical area or a non-critical area.

[0101] Here, the image division branch 72f includes a pooling layer 72L 11 , convolutional layer 72L 12 , Pooling layer 72L 13 , convolutional layer 72L 14 , Pooling layer 72L 15 , convolutional layer 72L 16 , convolutional layer 72L 17 , convolutional layer 72L 18 , bilinear interpolation layer 72L 19 , and output mask layer 72L 20 For example, the pooling layer 72L may include 11 The filter parameters may be 5*5, take 5 as the step size, and use the convolution layer 72L. 12 may use 512 1*1 convolution kernels, with a step size of 1, and a pooling layer 72L 13 The filter parameters may be 10*10, and the step size is 10. 14 may use 512 1*1 convolution kernels, with a step size of 1, and a pooling layer 72L 15 The filter parameters may be 14*14, take 14 as the step size, and use the convolution layer 72L 16 may use 512 1*1 convolution kernels, taking 1 as the step size, and the convolution layer 72L 17may use 512 3*3 convolution kernels, take a step size of 1, and convolution layer 72L 18 may use two 1*1 convolution kernels, with a step size of 1. In the image segmentation branch, a multi-size pooling operation is performed on the feature maps obtained from the public network layer, and then these feature maps are connected and bilinearly interpolated to obtain a segmentation mask map (Mask) of the image by newly interpolating the feature maps into an image of the same size as the input image, and this segmentation mask map can then be determined as the segmented image corresponding to the input image.

[0102] For example, the computer device may determine each of the clipped images in the set of clipped images shown in Fig. 7 as the processing target image. For example, the computer device may input the clipped image 70P1 to the image recognition and segmentation model 70W shown in Fig. 7. The image classification branch 71f performs recognition processing on the clipped image 70P1 to obtain a predicted classification confidence (e.g., 0.9) corresponding to the clipped image 70P1. At the same time, the image segmentation branch 72f removes and processes non-critical portions of the clipped image 70P1 to obtain a segmented image corresponding to the clipped image 70P1.

[0103] Similarly, the computer device may input the cropped image 70P2 to the image recognition segmentation model 70W. The image classification branch 71f performs recognition processing on the cropped image 70P2 to obtain a predicted classification confidence (e.g., 0.3) corresponding to the cropped image 70P2. At the same time, the image segmentation branch 72f removes non-critical portions of the cropped image 70P2 to obtain a segmented image corresponding to the cropped image 70P2.

[0104] The computer device may input the cropped image 70P3 into the image recognition segmentation model 70W. The image classification branch 71f performs recognition processing on the cropped image 70P3 to obtain a predicted classification confidence (e.g., 0.87) corresponding to the cropped image 70P3. At the same time, the image segmentation branch 72f removes non-critical portions of the cropped image 70P3 to obtain a segmented image corresponding to the cropped image 70P3.

[0105] Because the predicted classification confidence corresponding to the cropped image 70P1 and the predicted classification confidence corresponding to the cropped image 70P3 are both greater than a confidence threshold (e.g., 0.8), the computer device can determine that the cropped image 70P1 and the cropped image 70P3 both belong to the classification category of the critical region. In this case, the computer device may determine that the segmented image corresponding to the cropped image 70P1 (e.g., segmented image 71P1 shown in FIG. 7) and the segmented image corresponding to the cropped image 70P3 (e.g., segmented image 71P3 shown in FIG. 7) are collectively the segmentation results associated with the critical region.

[0106] As can be seen, the image recognition segmentation model 70W can simultaneously output pixel-level image segmentation results and macro-level image detection results, which means that the embodiment of the present application can fully utilize the macro-image information for the classification task and the micro-pixel information for the segmentation task to learn more comprehensive features of expressive ability. That is, multi-tasking can improve the overall effect of the algorithm flow, output two types of abilities, and ensure extremely high accuracy.

[0107] In step S104, in the set of adjustment boxes, M adjustment boxes corresponding to the M divided images are corrected based on the coordinate positions of the M divided images, thereby obtaining M correction boxes to indicate critical areas in the image to be detected.

[0108] Specifically, when the computer device obtains M divided images as a result of the division, it needs to determine adjustment boxes corresponding to each of these M divided images in the set of adjustment boxes, and then, by performing a correction process on the coordinate positions of the determined adjustment boxes based on the coordinate positions of the divided images, it is possible to obtain M correction boxes to indicate critical areas in the image to be detected.

[0109] 2, the computer device can determine that the adjustment box corresponding to segmented image P1 in the adjustment box set is adjustment box 22B1, and can obtain modification box 23B1 by modifying adjustment box 22B1 based on the coordinate position of segmented image P1. Similarly, the computer device can determine that the adjustment box corresponding to segmented image P2 in the adjustment box set is adjustment box 22B3, and can obtain modification box 23B2 by modifying adjustment box 22B3 based on the coordinate position of segmented image P2. This means that after the computer device in this embodiment performs the modification process of the adjustment box set based on the two segmented images in the segmentation result, it can accurately obtain two bounding boxes (i.e., modification boxes) for indicating the critical regions in image 200p, which means that there are faces of two different service objects in image 200p.

[0110] In this embodiment, an end-to-end closed-loop design for image detection and segmentation is completed by a cascade model set that links together parts such as an image detection model, a deduplication rule based on shape adaptive maximum suppression, an image shift adjustment model, and an image recognition segmentation model. This image detection method comprehensively considers the inherent relationship between the deduplication and shift adjustment process (including deduplication and position correction) and the two tasks of image detection and image segmentation, and places importance on the index of the accuracy of determining the position of the bounding box where the critical part is located, thereby obtaining a modified box that can more accurately indicate the critical part of the image to be detected, and further effectively improving the accuracy of image detection.

[0111] Further, please refer to Fig. 8. Fig. 8 is a schematic diagram of an image detection framework provided in an embodiment of the present application. As shown in Fig. 8, an iterative localization-type image detection and segmentation closed-loop architecture based on deep learning is mainly proposed in the embodiment of the present application. This architecture may include a rapid image detection stage, an iterative shift correction localization box stage, and a multi-task image detection and segmentation stage.

[0112] Here, in the rapid image detection stage, when the computer equipment acquires the image to be detected, it may input the image to be detected into the image detection model 81W shown in Figure 8, and further, the image detection model 81W can obtain location determination boxes (i.e., candidate boxes) where a large number of pseudo-critical areas are located from the image to be detected with high recall, thereby providing rapid and good pre-screening for the subsequent image detection process, that is, by image detecting the critical areas of the image to be detected, a candidate box set containing multiple candidate boxes is obtained.

[0113] Because there may be candidate boxes overlapping the candidate box set, in the iterative shift correction position determination box stage, the computer device may perform smart deduplication of the candidate box set using a deduplication rule based on shape-adaptive local maximum suppression. This allows the most effective candidate boxes to be retained and reduces the amount of subsequent calculations. Furthermore, the computer device may input the candidate box set (i.e., the target box set) after the deduplication process and the target image together into an image shift adjustment model 82W, and iteratively perform fine-tuning of the position shifts (i.e., position correction) of H (H is a positive integer) candidate boxes in the target box set until H bounding boxes (i.e., adjustment boxes) indicating the optimal positions of each critical region are obtained. To improve the efficiency of subsequent detection, the computer device may perform the deduplication process again on the H candidate boxes to obtain a set of adjustment boxes and a set of cropped images corresponding to the adjustment box set. Here, each cropped image in the set of cropped images is obtained by cropping the target image based on the adjustment boxes in the adjustment box set.

[0114] In the multi-task image detection and segmentation step, the computer device needs to perform image classification and image segmentation again on the remaining bounding boxes (i.e., the adjustment boxes in the adjustment box set), and can output bounding boxes in which the critical regions of the classification set are located, and can also output pixel-level segmented images with the background removed. For example, the computer device may input N cropped images in the cropped image set into the image recognition segmentation model 83W shown in FIG. 8 . Furthermore, the image classification branch and the image segmentation branch in the image recognition segmentation model 83W can determine that the segmented images whose classification category belongs to the critical region are included. Furthermore, a segmentation result associated with the critical region can be obtained. Here, the segmentation result may include M segmented images, where M is a positive integer less than or equal to N, and N is a positive integer less than or equal to H.

[0115] Finally, the computer device may perform a correction process on the adjustment boxes corresponding to the M divided images in the adjustment box set, thereby obtaining a correction box set, which may include M correction boxes for indicating critical regions in the detection target image.

[0116] As can be seen, the input of the detection system corresponding to the detection framework shown in FIG. 8 may be a target image to be detected, and the output of the system may be whether or not a critical feature (e.g., a face) is present in the current target image. If a critical feature is present in the current target image, the system outputs a bounding box (i.e., a correction box) in which the critical feature is located and a pixel-level foreground image (i.e., a segmented image with the background removed) corresponding to each correction box. Due to the diversity of the critical feature itself, changes in viewing angle, lighting conditions, occlusion, and the influence of complex background, the image detection model 81W, the duplicate removal rule, the image shift adjustment model 82W, and the image recognition segmentation model 83W in the embodiment of the present application are all customized based on the critical feature. Therefore, the framework according to the embodiment of the present application can be customized to accommodate more image object detection tasks. Image detection based on this framework satisfies the need for rapid detection of complex critical features in the target image, and has the advantages of low implementation / execution costs, high recognition recall / accuracy, and excellent online performance.

[0117] Further, please refer to FIG. 9. FIG. 9 is a schematic diagram of the flow of the image detection model training method proposed in the embodiment of the present application. The method may be executed by a computer device having a model training function. The computer device may be a terminal device (e.g., any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, e.g., terminal device 100a) or a server having a model training function (e.g., server 10F shown in FIG. 1 above). This is not limited here. The method may include at least the following steps S201 to S203.

[0118] In step S201, a first sample image for training a first detection model and a first sample label for indicating the actual classification of the first sample image are obtained.

[0119] Here, the network structure of the first detection model may refer to the network structure shown in FIG. 4 above. Specifically, it may include two convolutional layers, two pooling layers, two normalization layers, one special network layer (e.g., network layer 4L7), and a final output layer (label layer). The first sample image is obtained by preprocessing a critical region of an original sample image. For example, if the critical region is a face, the original sample image may be a face image in a face database (e.g., Annotated Facial Landmarks in the Wild, also known as the face dataset). After obtaining the original sample image, the computer device may generate a face / non-face training set by local block division, inversion, Gaussian blurring, etc.

[0120] In step S202, the first detection model is called to perform image detection on the first sample image, thereby obtaining a predicted classification confidence of the first sample image for the critical region.

[0121] In step S203, a first detection model is trained based on the predicted classification reliability of the first sample image and the actual classification of the first sample image to obtain an image detection model for detecting the critical portion of the detection target image.

[0122] Specifically, the computer device may determine a model loss of the first detection model by performing a loss calculation process between the predicted classification confidence of the first sample image and the actual classification of the first sample image, and may further train the first detection model based on the model loss of the first detection model to obtain a first model training result. If the first model training result indicates that the trained first detection model satisfies the first model convergence condition, the computer device may select the first detection model that satisfies the first model convergence condition as a second detection model, and may further perform a structural analysis of the network structure of the second detection model to obtain an analysis result. Based on the analysis result and the second detection model, the computer device may generate an image detection model for image detection of critical regions in the detection target image.

[0123] If the network structure of the first detection model designed in the embodiment of the present application is the first network structure (i.e., the network structure when the network layer 4L7 shown in Figure 4 is a fully connected layer), the computer device may train the first detection model, and when a first detection model (i.e., a second detection model) that satisfies the model convergence condition is obtained, the second detection model may directly be used as the final image detection model.

[0124] Here, the goal of training the first detection model is to minimize a loss function, i.e., to minimize the difference between the predicted label and the true label by training the network, and this difference is called the loss. There are many types of loss functions, and for classification tasks, in embodiments of the present application, a loss function that combines a cross-entropy loss function and a regularization term can be used. Specifically, see Equations 6 to 8 below.

number

[0125] Here, N is the number of samples in each iteration, i.e., the batch size, and L(y i ,p i) belongs to the loss value of the i-th sample image among the first sample images, λ controls the weight of the regularization term, R is the regularization term, and w is the weight value.

number

[0126] Here, C refers to the total number of classification categories (here, 2 can be used as an example, i.e., critical and non-critical parts), and y ij is the true classification confidence that the i-th sample image among the first sample images determined from the first sample label belongs to the classification category j (for example, the classification category called critical part), and p ij is the predicted category confidence that the ith sample image belongs to the classification category j (for example, the classification category of critical region) determined based on the first detection model.

number

[0127] where K is the number of weights and w k is the kth weight out of K weights.

[0128] As can be seen, the cross-entropy loss function measures the difference between different distributions, and has a good measuring effect on the two tasks of whether or not they belong to the critical region in the embodiment of this application. The regularization loss function represents the sum of squares of the weights, which can effectively restrict the range of the weights and improve the versatility of the model.

[0129] It should be understood that when a first sample image is input into a first detection model whose network structure belongs to a first network structure, the computer device can determine the model loss of the first detection model based on the above Equations 6 to 8, and further train the first detection model based on the model loss and a model convergence condition corresponding to the first detection model (i.e., the first model convergence condition), thereby obtaining a first model training result. The first model convergence condition here refers to whether the current model loss is equal to or less than a predetermined threshold, or whether the number of training steps is equal to or greater than a predetermined threshold.

[0130] If the first model training result indicates that the first detection model after training satisfies the first model convergence condition, the computer device may use the first detection model that satisfies the first model convergence condition as the second detection model, and may further use the second detection model directly as an image detection model for image detection of critical areas in the detection target image.

[0131] Optionally, if the network structure of the first detection model designed in the embodiments of the present application is the first network structure, the finally obtained second detection model has a fully connected layer, and the input sample image needs to be scanned in a sliding window manner in the image detection process, which means that the entire process takes a very long time. Here, the specific formula for the number of classifications can be seen in Equation 9 below.

number

[0132] where L is intended to represent the length of the input image, W is intended to represent the width of the input image, and S is intended to represent the sliding step size of the sliding window.

[0133] Based on this, assuming that the input sample image is 800 × 600 and the sliding step size used is 32, the number of classifications for the input sample image can be determined to be 450 according to Equation 9 above. Furthermore, considering that the input sample image is scaled to different sizes before being input to the network to detect critical regions of different sizes, the number of required classifications increases further. In practice, when performing image detection based on a sliding window, there is an overlap between adjacent windows. This overlapping image is calculated repeatedly during the sliding calculation process, resulting in unnecessary computational resource waste. To overcome this problem, in the present embodiment, a second detection model having a fully connected layer is obtained through training, and then the second detection model having a fully connected layer is converted into a fully convolutional network, and the converted second detection model is determined as the final image detection model.

[0134] For example, if the analysis structure indicates that the second detection model has a fully connected layer, the computer device may replace the fully connected layer in the second detection model with a first convolutional layer in which a sliding step size is arranged, and further obtain a second convolutional layer by assigning a value to the first convolutional layer based on the network parameters of the fully connected layer, and determine the second detection model including the second convolutional layer as an image detection model for image detection of a critical portion of a detection target image. Before conversion to a fully convolutional network, the model allows input of pictures of a fixed image size (e.g., 48 × 48 × 3), and after conversion to a fully convolutional network, the model allows input of pictures of any image size.

[0135] Optionally, when the model result of the first detection model designed in the embodiment of the present application is a second network structure (i.e., a network structure in which the network layer 4L7 shown in FIG. 4 is a convolutional layer), the computer device may determine that the analysis result of the first detection model (i.e., the second detection model) that satisfies the first model convergence condition indicates that the second detection model is a fully convolutional network. In this case, the computer device can perform image detection on images of any size. In this case, the computer device may directly determine the second detection model belonging to the fully convolutional network as the final image detection model.

[0136] In other words, the image detection model in the embodiment of the present application (for example, image detection model 81W shown in FIG. 8 above) may include three cases. One is a case where a first detection model belonging to a first network structure is directly trained. Another is a case where a first detection model belonging to a first network structure is trained and then fully convolutional network conversion is performed. And yet another is a case where a first detection model belonging to a second network structure is directly trained. Here, the training process of the image detection model is not limited. The trained image detection model of the present application is used to perform step S101 in the embodiment corresponding to FIG. 3 above, in which, when a detection target image is acquired, a set of candidate boxes is acquired by image detection of critical portions of the detection target image using the image detection model.

[0137] In the present embodiment, an image detection model based on a fully convolutional network is proposed, which can detect images of any size using a sliding window and detect critical regions from the image as much as possible with a very high coverage rate. This design is in line with the characteristics of the task, can detect critical regions of any size in images of any size in a very short time, provide good pre-screening for subsequent processes, improve the execution efficiency of the entire system, and avoid a large amount of unnecessary calculations.

[0138] Further, please refer to FIG. 10. FIG. 10 is a schematic diagram of the flow of a method for training an image shift adjustment model proposed in an embodiment of the present application. The method may be executed by a computer device having a model training function. The computer device may be a terminal device (e.g., any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, e.g., the terminal device 100a), or a server having a model training function (e.g., the server 10F shown in FIG. 1 above). This is not limited thereto. The method may include at least the following steps S301 to S305.

[0139] In step S301, a second sample image for training an initial shift adjustment model and a second sample label for indicating the actual coordinate position where the critical portion of the second sample image is located are obtained.

[0140] In order to achieve a sufficient number of shift mode classification capabilities and reduce the computational complexity as much as possible, in the embodiment of the present application, a network structure (e.g., A-Net) of an initial shift adjustment model may be constructed based on the network structure (e.g., D-Net) of an image detection model. The initial shift adjustment model may refer to the network structure of Figure 6 above, and may specifically include four convolutional layers, three pooling layers, one fully connected layer, and a final output layer (label layer).

[0141] If the critical feature belongs to a face, there are many publicly available face-labeled datasets available online for face detection, and you can directly request a download from the creator. For the labeled face dataset, you also need to extract training data. Specifically, you can obtain the correct face locations based on the face labels, and then shift these faces by a certain amount (i.e., actual shift amounts) to generate shifted faces and corresponding shift parameters. In this embodiment, the shifted face images are divided into a training set, a validation set, and a test set. The training set can be used as a second sample image for further training. The second sample label of the second sample image indicates the actual coordinate position of the second sample image, i.e., the coordinate position determined based on the shift parameters and the coordinate position of the face before shifting.

[0142] In step S302, the initial shift adjustment model is called and shift prediction processing of the second sample image is performed to obtain predicted regression parameters of the second sample image.

[0143] In step S303, the predicted coordinate position of the second sample image is acquired by performing an adjustment process for the coordinate position of the second sample image based on the predicted regression parameters.

[0144] Specifically, the computer device may determine the predicted coordinate position of the second sample image by referring to the above equation 5 and performing an adjustment process on the corresponding coordinates (e.g., a certain vertex coordinate, width, height) at the coordinate position of the second sample image based on each predicted deviation amount in the predicted regression parameters of the second sample image.

[0145] In step S304, an initial shift adjustment model is trained based on the predicted coordinate position and the actual coordinate position to obtain a second model training result.

[0146] Specifically, the computer device may determine a model loss of the initial shift adjustment model based on the predicted coordinate position and the actual coordinate position indicated by the second sample label, and further may obtain a second model training result by training the initial shift adjustment model based on the model loss of the initial shift adjustment model.

[0147] Here, the initial shift adjustment model needs to predict the amount of shift between the region where the critical portion of the second sample image is located and the nearest bounding box (e.g., the predicted shift amount corresponding to the top left corner of the bounding box, the predicted shift amount corresponding to the height, and the predicted shift amount corresponding to the width). The learning goal is set as a regression problem. The model loss of the initial shift adjustment model may be determined based on the sample loss of each sample in the second sample image. The sample loss of one sample may use Euclidean loss, specifically, see Equation 10 below.

number

[0148] where i denotes the i-th sample of the second sample image,

number

number

number

[0149] In step S305, if the second model training result indicates that the initial shift adjusted model after training satisfies the second model convergence condition, the initial shift adjusted model that satisfies the second model convergence condition is determined as the image shift adjusted model.

[0150] The second model convergence condition here refers to whether the current model loss is below a predetermined threshold or the number of training steps is above a predetermined threshold. It can be understood that the network structure of the image shift adjustment model (e.g., A-Net) can also be a critical region location correction network. This network is used to correct the critical region location by using the regression parameters of the bounding box where the critical region is located after feature extraction of the target image.

[0151] In this embodiment, a trained deep shift prediction model (e.g., image shift adjustment model 82W shown in FIG. 8 above) is used to predict the shift of the set of candidate boxes after de-duplication obtained in the previous step (e.g., step S102 in the embodiment corresponding to FIG. 3 above), and then iteratively perform position correction to achieve more accurate positioning. The adjusted boxes obtained in this configuration have extremely high positioning accuracy, and can particularly accurately determine the positions of extremely small critical regions and effectively distinguish adjacent or locally overlapping critical regions.

[0152] Further, please refer to FIG. 11. FIG. 11 is a schematic diagram of the flow of a method for training an image recognition segmentation model proposed in an embodiment of the present application. The method may be executed by a computer device having a model training function. The computer device may be a terminal device (e.g., any one of the terminal devices in the terminal device cluster shown in FIG. 1 above, e.g., terminal device 100a) or a server having a model training function (e.g., server 10F shown in FIG. 1 above). This is not limited here. The method may include at least the following steps S401 to S405.

[0153] In step S401, sample data for training an initial recognition segmentation model and sample labels corresponding to the sample data are obtained.

[0154] In order to better capture image information of critical regions at the macro and micro levels, the initial recognition segmentation model in the embodiment of the present application may incorporate a residual module and a pyramid pooling module to construct a network structure (e.g., S-Net) for the initial recognition segmentation model. For example, see the network structure shown in FIG. 7. Specifically, the initial recognition segmentation model may include a public network layer 70G, an image classification branch 71f, and an image segmentation branch 72f.

[0155] Here, the network tasks include an image classification task and an image segmentation task, but the training sets for these two tasks may be incompatible. This is because, in the image segmentation task, if a sample image of a non-critical region is introduced, the segmentation mask of this sample image of the non-critical region is set to the non-critical region of the entire map. In this case, the loss function during the training process increases several times, significantly affecting the accurate segmentation of the critical region. In consideration of these factors, in this embodiment, a stepwise training method is adopted for these two tasks. That is, the sample data here may include a third sample image for training the image segmentation branch and a fourth sample image for training the image classification branch. The sample labels here may include a third sample label corresponding to the third sample image (i.e., representing the actual mask image) and a fourth sample label corresponding to the fourth sample image (i.e., representing the actual classification).

[0156] Since the network involves the label compatibility problem of the two tasks, a multi-stage training method can be adopted in the embodiment of the present application. Therefore, in the embodiment of the present application, loss functions are defined for the two tasks respectively. Specifically, the loss function for the image segmentation task can be referred to as Equation 11 below.

number

[0157] where N can represent the number of training samples in one iteration in the third sample image, H is the height of the mask image, W is the width of the mask image, T is the number of classification categories (e.g., 2), and f ijwh can be used to represent the predicted classification confidence (i.e., the second confidence) of the ith sample for the jth classification at pixel location (w,h), and y ijwh can be used to represent the actual classification confidence (ie, the first confidence) of the ith sample for the jth classification at pixel location (w,h).

[0158] Specifically, the loss function for the image classification task can be calculated by referring to Equation 12 below.

number

[0159] where N can represent the number of training samples in one iteration in the fourth sample image, T is the number of classification categories (e.g., 2), and f ij can be used to represent the predicted classification confidence of the ith sample for the jth classification (i.e., sample prediction confidence), and y ijwh can be used to represent the actual classification confidence of the ith sample for the jth classification.

[0160] In step S402, a first recognition division model is obtained by training the public network layer and image division branch in the initial recognition division model based on a third sample image in the sample data and a third sample label for indicating the actual mask image of the third sample image.

[0161] In step S403, a second recognition division model is obtained by training the public network layer and image classification branch in the first recognition model based on a fourth sample image in the sample data and a fourth sample label for indicating the actual classification of the fourth sample image.

[0162] In step S404, a third recognition segmentation model is obtained by training the public network layer and the image segmentation branch in the second recognition segmentation model according to the third sample image and the third sample label.

[0163] Specifically, after completing the execution of step S403, the computer device needs to repeat the training of the public network layer and the image segmentation branch in the second recognition model based on the third sample image and the third sample label at a smaller learning rate until the loss is stabilized.

[0164] For example, the computer device may call the image segmentation branch of the second recognition division model, remove and process non-critical portions of the third sample image, thereby obtaining a predicted mask image corresponding to the third sample image, and then scan pixels of the third sample image to determine the scanned pixel as the target pixel. In this case, the computer device may determine the confidence that the target pixel is in the actual mask image indicated by the third sample label as a first confidence, and determine the confidence that the target pixel is in the predicted mask image as a second confidence. The computer device may then determine the model loss of the second recognition division model by performing a loss calculation process for the first confidence and the second confidence based on the above Equation 11. Next, the computer device may train the public network layer and the image segmentation branch of the second recognition division model based on the model loss of the second recognition division model to obtain a third model training result. If the third model training result indicates that the trained second recognition division model satisfies the first branch convergence condition of the third model convergence condition, the second recognition division model that satisfies the first branch convergence condition may be determined as the third recognition division model. The first branch convergence condition here can refer to the model loss reaching stability.

[0165] In step S405, the public network layer and image segmentation branch in the third recognition segmentation model are locked, and the image classification branch in the locked third recognition segmentation model is trained based on the fourth sample image and the fourth sample label, thereby obtaining an image recognition segmentation model.

[0166] Specifically, the computer device may lock the public network layer and the image segmentation branch in the third recognition division model, and then determine the locked third recognition division model as the fourth recognition division model. In this case, the computer device may call the fourth recognition division model and perform recognition processing on a fourth sample image using the image classification branch in the fourth recognition division model to determine a sample prediction confidence corresponding to the fourth sample image. The computer device may also determine the model loss of the fourth recognition division model by performing a loss calculation process between the actual classification indicated by the fourth label and the sample prediction confidence according to Equation 12 above. In this case, the computer device may train the image classification branch in the fourth recognition division model based on the model loss of the fourth recognition division model to obtain a fourth model training result. If the fourth model training result indicates that the trained fourth recognition division model satisfies the second branch convergence condition in the third model convergence condition, the fourth recognition division model that satisfies the second branch convergence condition may be determined as the image recognition division model. Here, the second branch convergence condition may also refer to the model loss reaching stability. The trained image recognition and segmentation model of the present application is intended to execute step S103 in the embodiment corresponding to Figure 3 above, in which the image recognition and segmentation model performs recognition and segmentation processing on each of the N clipped images to obtain segmentation results associated with the critical parts.

[0167] In the present embodiment, this step-by-step training method allows the public network layer in the finally obtained image recognition segmentation model (e.g., image recognition segmentation model 83W shown in FIG. 8 above) to effectively represent the input image features for both tasks. These features are sent to their respective branches and can successfully perform each task. Based on this, the present embodiment combines these two tasks into a single network and provides step-by-step joint training. This allows the network to learn critical features that balance the two tasks. The inherent relationship between these two tasks also forces the network to learn features with better representation ability during the training process, achieving a mutual amplification effect. Furthermore, at the specific structural level of the image recognition segmentation model, the present embodiment achieves a good balance between segmentation effectiveness and execution speed through structural optimization. The overall network complexity is not high, and the execution efficiency of a single network can reach 25 frames per second. In the actual application process of the model, the image segmentation branch of the image recognition segmentation model can use the image information in the cropped image to clear the background or other non-key critical areas in the cropped image, making the critical areas in the segmented image clearer and purer, and outputting pixel-level image information (e.g., face information). In addition, the image classification branch of the image recognition segmentation model can perform image detection again to predict the classification category of the adjustment box corresponding to the cropped image, further improving the accuracy of image detection.

[0168] Further, please refer to FIG. 12. FIG. 12 is a schematic diagram of the configuration of an image processing device provided in an embodiment of the present application. As shown in FIG. 12, the image processing device 1 may include an image detection module 11, a deduplication adjustment module 12, a recognition and division module 13, a correction module 14, a first sample acquisition module 15, a sample detection module 16, a first training module 17, a second sample acquisition module 18, a sample prediction module 19, a predicted coordinate determination module 20, a second training module 21, an adjustment model determination module 22, a third sample acquisition module 23, a third training module 24, a fourth training module 25, a fifth training module 26, and a sixth training module 27.

[0169] When the image detection module 11 acquires an image to be detected, it performs image detection of critical parts of the image to be detected to acquire a set of candidate boxes, and the set of candidate boxes includes candidate boxes for marking the image detection results.

[0170] The deduplication adjustment module 12 obtains a set of boxes to be processed by performing deduplication processing on candidate boxes in a set of candidate boxes based on the image to be detected, obtains adjustment boxes by correcting the positions of the candidate boxes in the set of boxes to be processed, and generates a set of adjustment boxes and a set of cut-out images corresponding to the set of adjustment boxes based on the adjustment boxes, where the set of cut-out images includes N cut-out images (N is a positive integer), and one cut-out image is obtained by cutting out the image to be detected based on one adjustment box in the set of adjustment boxes.

[0171] Here, the deduplication adjustment module 12 includes a first deduplication unit 121, an adjustment unit 122, a second deduplication unit 123, and a fourth determination unit .

[0172] The first deduplication unit 121 obtains a target box set by performing deduplication processing on the candidate boxes in the candidate box set based on the shape adaptive maximum suppression deduplication rule, where the target box set includes candidate boxes Xi, where i is a positive integer less than or equal to H, and H indicates the total number of candidate boxes in the target box set.

[0173] Here, the candidate box set is obtained by invoking an image detection model to detect critical areas in the target image, and the candidate box set includes K candidate boxes (K is a positive integer), and the image detection model is further used to determine a predicted classification confidence for each of the K candidate boxes.

[0174] The first deduplication unit 121 includes a first clipping subunit 1211 , a sorting subunit 1212 , a second determining subunit 1213 , a deduplication subunit 1214 and an expanding subunit 1215 .

[0175] The first clipping subunit 1211 obtains K clipping boxes by performing a clipping process on each of the K candidate boxes according to the image clipping rate set by the shape adaptive maximum suppression duplicate elimination rule.

[0176] The sorting sub-unit 1212 performs a sorting process on the K clipping boxes according to the K predicted classification confidences to obtain a sorting result.

[0177] The second determination subunit 1213 determines the clipping box with the highest predicted classification confidence in the sorted result as the first clipping box, and determines the (K-1) clipping boxes other than the first clipping box in the sorted result as the set to be filtered.

[0178] The deduplication subunit 1214 obtains a set of reserved boxes by performing a deduplication process on the K clipping boxes based on the overlapping degree between the first clipping box and each clipping box in the set to be filtered.

[0179] Here, the deduplication subunit 1214 specifically: Retaining the first clipping box, and determining the overlap between the first clipping box and each clipping box in the filtering target set; If there is an overlapping clipping box in the filtering target set whose overlap degree is greater than the overlap degree threshold, filtering the overlapping clipping box in the filtering target set; In the filtering target set after filtering, the clipping box having the maximum predicted classification reliability is set as a second clipping box, and the clipping boxes other than the second clipping box in the filtering target set after filtering are set as a new filtering target set; The second clipping box is reserved, and the deduplication process of the new filtering target set is continued based on the degree of overlap between the second clipping box and each clipping box in the new filtering target set until the filtering target set after the deduplication process is empty, and the reserved clipping box is determined as a reserved box set, and the reserved box set includes the first clipping box and the second clipping box.

[0180] The expansion subunit 1215 obtains a set of boxes to be processed by respectively performing image expansion processing on each of the clipping boxes in the set of reserved boxes according to the image clipping ratio.

[0181] Here, for specific implementations of the first clipping subunit 1211, the sorting subunit 1212, the second determining subunit 1213, the deduplication subunit 1214, and the expansion subunit 1215, please refer to the description of the deduplication process of the candidate box set in the embodiment corresponding to Figure 5. Here, further description will be omitted.

[0182] The adjustment unit 122 calls an image shift adjustment model to calculate the candidate box X i By correcting the position of the adjustment box Y i and adjustment box Y i and the corresponding cropped images are obtained.

[0183] Here, the adjustment unit 122 includes a first prediction subunit 1221, a first adjustment subunit 1222, a third determination subunit 1223, a second clipping subunit 1224, a second prediction subunit 1225, a second adjustment subunit 1226, and a fourth determination subunit 1227.

[0184] The first prediction sub-unit 1221 calls an image shift adjustment model to generate a candidate box X according to the target image. i The first regression parameters are obtained by predicting the deviation of

[0185] The first adjusting subunit 1222 determines a candidate box X based on the first regression parameters. i By correcting the position of the candidate box X i Get the first adjustment box corresponding to

[0186] The third determining subunit 1223 determines whether the first regression parameter is in the regression parameter threshold range. i The first adjustment box corresponding to the candidate box X i Corresponding adjustment box Y i It is determined as follows.

[0187] The second cutout subunit 1224 is connected to the adjustment box Y i The adjustment box Y is created by cutting out the detection target image based on the coordinate position of i Obtain the corresponding cropped image.

[0188] If the first regression parameter is not within the regression parameter threshold range, the second prediction subunit 1225 calls the image shift adjustment model, and calculates a candidate box X according to the target image. i The second regression parameters are obtained by predicting the deviation of the first adjustment box corresponding to

[0189] The second adjusting subunit 1226 adjusts the candidate box X based on the second regression parameters. i By correcting the position of the first adjustment box corresponding to i Get the second adjustment box corresponding to

[0190] The fourth determining subunit 1227 determines whether the second regression parameter is in the regression parameter threshold range. i The second adjustment box corresponding to the candidate box X i Corresponding adjustment box Y i It is determined as follows.

[0191] Here, for specific implementations of the first prediction subunit 1221, the first adjustment subunit 1222, the third determination subunit 1223, the second clipping subunit 1224, the second prediction subunit 1225, the second adjustment subunit 1226, and the fourth determination subunit 1227, please refer to the description of the candidate box shift adjustment process in the embodiment corresponding to Figure 3 above, and further description will be omitted here.

[0192] When the second deduplication unit 123 acquires the H adjustment boxes, it performs deduplication processing on the H adjustment boxes based on the deduplication rules.

[0193] The fourth determination unit 124 determines the adjustment box after the deduplication process as an adjustment box set corresponding to the candidate box set, and determines the clipped images corresponding to the adjustment box after the deduplication process as a clipped image set corresponding to the adjustment box set.

[0194] For specific implementations of the first deduplication unit 121, the adjustment unit 122, the second deduplication unit 123, and the fourth determination unit 124, please refer to the description of step S102 in the embodiment corresponding to Figure 3 above. No further description will be given here.

[0195] The recognition and division module 13 performs recognition and division processing on each of the N cutout images to obtain a division result associated with the critical portion, and the division result includes M (M is a positive integer less than or equal to N) cutout images.

[0196] Here, the recognition division module 13 includes a fifth determination unit 131, a first recognition unit 132, a removal unit 133 and a sixth determination unit .

[0197] The fifth determination unit 131 determines an image to be processed from the N cutout images.

[0198] The first recognition unit 132 calls an image recognition division model, and recognizes and processes the target image according to the image classification branch in the image recognition division model, thereby obtaining a predicted classification confidence corresponding to the target image.

[0199] The removal unit 133 removes non-critical portions of the target image according to the image segmentation branch in the image recognition segmentation model, thereby obtaining a segmented image corresponding to the target image.

[0200] If the predicted classification reliability corresponding to the image to be processed is greater than the reliability threshold, the sixth determination unit 134 determines the segmented image corresponding to the image to be processed as the segmentation result associated with the critical region.

[0201] Here, for the specific implementation of the fifth determination unit 131, the first recognition unit 132, the removal unit 133, and the sixth determination unit 134, please refer to the description of step S103 in the embodiment corresponding to Figure 3 above. Here, further description will be omitted.

[0202] The correction module 14 performs a correction process on the adjustment boxes corresponding to the M divided images in the set of adjustment boxes, thereby obtaining M correction boxes for indicating critical portions in the detection target image.

[0203] The first sample acquisition module 15 acquires a first sample image for training a first detection model and a first sample label for indicating the actual classification of the first sample image, and the first sample image is obtained by preprocessing the critical part of the original sample image.

[0204] The sample detection module 16 invokes a first detection model to image detect the first sample image to obtain a predicted classification confidence of the first sample image for the critical region.

[0205] The first training module 17 obtains an image detection model for image detection of critical areas in the target image by training a first detection model based on the predicted classification confidence of the first sample image and the actual classification of the first sample image.

[0206] Here, the first training module 17 includes a first determining unit 171 , a first training unit 172 , a second determining unit 173 and a third determining unit 174 .

[0207] The first determining unit 171 determines a model loss of the first detection model by performing a loss calculation process between the predicted classification confidence of the first sample image and the actual classification of the first sample image.

[0208] The first training unit 172 obtains a first model training result by training a first detection model based on the model loss of the first detection model.

[0209] If the first model training result indicates that the first detection model after training satisfies the first model convergence condition, the second determination unit 173 determines the first detection model that satisfies the first model convergence condition as the second detection model.

[0210] The third determination unit 174 obtains the analysis result by performing a structural analysis of the network structure of the second detection model, and generates an image detection model for image detection of the critical part of the image to be detected based on the analysis result and the second detection model.

[0211] Here, the third determining unit 174 includes an analyzing subunit 1741 , a replacing subunit 1742 , a value assigning subunit 1743 and a first determining subunit 1744 .

[0212] The analysis subunit 1741 performs a structural analysis of the network structure of the second detection model to obtain the analysis result.

[0213] If a fully connected layer exists in the network structure of the second detection model, the replacement subunit 1742 replaces the fully connected layer with a first convolution layer in which a sliding step size is arranged.

[0214] The value assignment subunit 1743 obtains the second convolution layer by assigning a value to the first convolution layer according to the network parameters of the fully connected layer.

[0215] The first determining subunit 1744 determines a second detection model including a second convolution layer as an image detection model for image detecting the critical portion of the detection target image.

[0216] Here, for specific implementations of the analysis subunit 1741, the replacement subunit 1742, the value assignment subunit 1743, and the first determination subunit 1744, please refer to the description of the network transformation of the second detection model in the embodiment corresponding to Figure 9 above. Here, further description will be omitted.

[0217] Here, for specific implementation forms of the first determination unit 171, the first training unit 172, the second determination unit 173, and the third determination unit 174, please refer to the description of step S203 in the embodiment corresponding to Figure 9 above. Here, further description will be omitted.

[0218] The second sample acquisition module 18 acquires a second sample image for training an initial shift adjustment model and a second sample label for indicating the actual coordinate position where the critical portion of the second sample image is located.

[0219] The sample prediction module 19 calls the initial shift adjustment model and performs shift prediction processing on the second sample image to obtain predicted regression parameters for the second sample image.

[0220] The predicted coordinate determination module 20 obtains the predicted coordinate position of the second sample image by performing position correction on the coordinate position of the second sample image based on the predicted regression parameters.

[0221] The second training module 21 obtains a second model training result by training an initial shift adjustment model based on the predicted coordinate position and the actual coordinate position.

[0222] If the second model training result indicates that the initial shift adjustment model after training satisfies the second model convergence condition, the adjustment model determination module 22 determines the initial shift adjustment model that satisfies the second model convergence condition as the image shift adjustment model.

[0223] The third sample acquisition module 23 acquires sample data for training an initial recognition segmentation model and sample labels corresponding to the sample data.

[0224] The third training module 24 obtains a first recognition segmentation model by training the public network layer and the image segmentation branch in the initial recognition segmentation model based on a third sample image in the sample data and a third sample label for indicating the actual mask image of the third sample image.

[0225] The fourth training module 25 obtains a second recognition segmentation model by training the public network layer and the image classification branch in the first recognition model based on a fourth sample image in the sample data and a fourth sample label for indicating the actual classification of the fourth sample image.

[0226] The fifth training module 26 obtains a third recognition segmentation model by training the public network layer and the image segmentation branch in the second recognition segmentation model based on the third sample image and the third sample label.

[0227] Here, the fifth training module 26 includes a prediction mask determining unit 261, a scanning unit 262, a sample reliability determining unit 263, a first loss determining unit 264, a second training unit 265, and a seventh determining unit 266.

[0228] The predicted mask determining unit 261 calls the second recognition segmentation model to remove and process the non-critical parts of the third sample image, thereby obtaining a predicted mask image corresponding to the third sample image.

[0229] The scanning unit 262 scans the pixels of the third sample image and determines the scanned pixels as pixels to be processed.

[0230] The sample reliability determination unit 263 determines the reliability that the pixel to be processed is in the actual mask image indicated by the third sample label as the first reliability, and determines the reliability that the pixel to be processed is in the predicted mask image as the second reliability.

[0231] The first loss determining unit 264 determines the model loss of the second recognition division model by performing a loss calculation process between the first reliability and the second reliability.

[0232] The second training unit 265 obtains a third model training result by training the public network layer and the image segmentation branch in the second recognition segmentation model based on the model loss of the second recognition segmentation model.

[0233] If the third model training results indicate that the second recognition division model after training satisfies the first branching convergence condition in the third model convergence condition, the seventh determination unit 266 determines the second recognition division model that satisfies the first branching convergence condition as the third recognition division model.

[0234] Here, for the specific implementation of the prediction mask determining unit 261, the scanning unit 262, the sample reliability determining unit 263, the first loss determining unit 264, the second training unit 265, and the seventh determining unit 266, please refer to the description of step S404 in the embodiment corresponding to Figure 10 above. Here, further description will be omitted.

[0235] The sixth training module 27 locks the public network layer and the image segmentation branch in the third recognition segmentation model, and obtains an image recognition segmentation model by training the image classification branch in the locked third recognition segmentation model based on the fourth sample image and the fourth sample label.

[0236] Here, the sixth training module 27 includes: a locking unit 271 , a second recognition unit 272 , a second loss determining unit 273 , a third training unit 274 and an eighth determining unit 275 .

[0237] The locking unit 271 locks the public network layer and the image segmentation branch in the third recognition division model, and determines the locked third recognition division model as the fourth recognition division model.

[0238] The second recognition unit 272 calls the fourth recognition division model and performs recognition processing on the fourth sample image to determine the sample prediction reliability corresponding to the fourth sample image.

[0239] The second loss determining unit 273 performs a loss calculation process between the actual classification indicated by the fourth label and the sample prediction reliability to determine the model loss of the fourth recognition division model.

[0240] The third training unit 274 obtains a fourth model training result by training an image classification branch in the fourth recognition division model based on the model loss of the fourth recognition division model.

[0241] If the fourth model training results indicate that the fourth recognition division model after training satisfies the second branching convergence condition in the third model convergence condition, the eighth determination unit 275 determines the fourth recognition division model that satisfies the second branching convergence condition as the image recognition division model.

[0242] Here, for the specific implementation of the locking unit 271, the second recognition unit 272, the second loss determination unit 273, the third training unit 274 and the eighth determination unit 275, please refer to the description of step S405 in the embodiment corresponding to Figure 10 above. Here, further description will be omitted.

[0243] Here, for specific implementations of the image detection module 11, the deduplication adjustment module 12, the recognition and segmentation module 13, the correction module 14, the first sample acquisition module 15, the sample detection module 16, the first training module 17, the second sample acquisition module 18, the sample prediction module 19, the predicted coordinate determination module 20, the second training module 21, the adjustment model determination module 22, the third sample acquisition module 23, the third training module 24, the fourth training module 25, the fifth training module 26, and the sixth training module 27, please refer to the description of the image processing method in the embodiment corresponding to Figures 3, 9, 10, and 11 above. Further description will be omitted here. Furthermore, a description of the beneficial effects of the same method will also be omitted.

[0244] Further, please refer to FIG. 13. FIG. 13 is a schematic diagram of a computer device provided in an embodiment of the present application. As shown in FIG. 13, the computer device 1000 may include at least one processor 1001, such as a CPU, at least one network interface 1004, a memory 1005, and at least one communication bus 1002. Here, the communication bus 1002 is used to realize communication connections between these components. Optionally, the network interface 1004 may include a standard wired interface or a wireless interface (e.g., a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may be at least one storage device located remotely from the processor 1001. As shown in FIG. 13, the memory 1005, which is a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application. Here, in some embodiments, the computer device may include a user interface 1003 shown in Fig. 13. For example, if the computer device is a terminal device (e.g., terminal device 100a) with a model training function shown in Fig. 1, the computer device may include the user interface 1003. Here, the user interface 1003 may include a display, a keyboard, etc.

[0245] In the computer device 1000 shown in FIG. 13, the network interface 1004 is mainly used for network communication, the user interface 1003 is mainly used for providing an input interface to a user, and the processor 1001 invokes a device control application stored in the memory 1005, a step of acquiring a detection target image, and performing image detection of a critical portion of the detection target image to acquire a candidate box set, the candidate box set including a candidate box for marking an image detection result; a step of obtaining a set of target boxes by performing a process of eliminating duplicates of candidate boxes in the set of candidate boxes based on the detection target image, obtaining adjustment boxes by correcting the positions of the candidate boxes in the set of target boxes, and generating a set of adjustment boxes and a set of clipped images corresponding to the set of adjustment boxes based on the adjustment boxes, wherein the set of clipped images includes N clipped images (N is a positive integer), and one clipped image is obtained by clipping the detection target image based on one adjustment box in the set of adjustment boxes; performing a recognition and segmentation process on each of the N number of clipped images to obtain a segmentation result associated with a critical portion, the segmentation result including M number of segmented images (M is a positive integer equal to or less than N); This method can be used to realize a step of obtaining M correction boxes to indicate critical areas in the image to be detected by performing correction processing on the adjustment boxes corresponding to the M divided images based on the coordinate positions of the M divided images in the adjustment box set.

[0246] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the image processing method described in the embodiment corresponding to Figures 3, 9, 10, and 11 above, and can also execute the image processing device 1 described in the embodiment corresponding to Figure 12 above. Further description will be omitted here, and the description of the beneficial effects of the same method will also be omitted.

[0247] In an embodiment of the present application, a computer-readable storage medium storing a computer program is further provided, the computer program including program instructions, which, when executed by a processor, realize the image processing methods provided in the steps of Figures 3, 9, 10, and 11. For details, please refer to the implementation forms provided in the steps of Figures 3, 9, 10, and 11 above. Further description will be omitted here.

[0248] The computer-readable storage medium may be an internal storage unit of the data transmission device or computer device provided in any one of the above-described embodiments, such as a hard disk or internal memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, or a flash card installed in the computer device. Furthermore, the computer-readable storage medium may include both an internal storage unit and an external storage device of the computer device. The computer-readable storage medium is intended to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store output data or data to be output.

[0249] In an embodiment of the present application, a computer program product is further provided, the computer program being stored in a computer-readable storage medium. A processor of the computer device reads the computer program from the computer-readable storage medium, and when the processor executes the computer program, the computer device performs the image processing method or apparatus described in each of the above embodiments. Further description is omitted here, and the beneficial effects of the same method are also omitted.

[0250] Terms such as "first" and "second" in the description, claims, and drawings of the embodiments of this application are intended to distinguish between different objects and not to describe a particular order. Furthermore, the term "comprises" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, and may optionally include unlisted steps or modules, or may optionally include other step units inherent to the process, method, apparatus, product, or device.

[0251] Those skilled in the art will recognize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized by electronic hardware, computer software, or a combination of both. To clearly explain the compatibility of hardware and software, the above description generally describes the configurations and steps of each example by function. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the invention. Professional engineers may use different methods to realize the described functions for each specific application, but such realizations should not be considered as going beyond the scope of this application.

[0252] The above are merely preferred embodiments of the present application, and of course, should not be used to limit the scope of the present application. Therefore, equivalent modifications according to the claims of the present application still fall within the scope of the present application. [Explanation of symbols]

[0253] 1. Image processing device 2. Bounding Box 3 Prediction classification reliability 10 Servers 11 Image detection module 12 Deduplication Coordination Module 13 Recognition and Segmentation Module 14 Correction Module 15 First sample acquisition module 16 Sample Detection Module 17 First Training Module 18 Second sample acquisition module 19 Sample Prediction Module 20 Prediction coordinate determination module 21 Candidate Box 21 Second Training Module 22 Adjustment Model Determination Module 23 Third Sample Acquisition Module 24 Third Training Module 25 Fourth Training Module 26 5th Training Module 27 Training Module 6 40 Image Detection Models 50 Candidate Box 51 Box 52 Candidate Box 60 Candidate Box 60 Image Shift Adjustment Model 61 Adjustment box 70 Image Recognition Segmentation Model 81 Image Detection Model 82 Image Shift Adjustment Model 83 Image Recognition Segmentation Model 100 Terminal Equipment 121 First Deduplication Unit 122 Adjustment Unit 123 Second Deduplication Unit 124 4th Decision Unit 131 5th Decision Unit 132 First Recognition Unit 133 units 134 6th Decision Unit 171 First Decision Unit 172 1st Training Unit 173 Second Decision Unit 174 Third Decision Unit 200 images 261 Prediction Mask Decision Unit 262 Scanning Unit 263 Sample Reliability Determination Unit 264 First Loss Determination Unit 265 2nd Training Unit 266 7th Decision Unit 271 Lock Unit 272 Second Recognition Unit 273 Second Loss Determination Unit 274 3rd Training Unit 275 8th Decision Unit 400 images 500 images 600 images 1000 Computer Equipment 1001 processor 1002 communication bus 1003 User Interface 1004 Network Interface 1005 memory 1211 subunit 1212 Sort Subunit 1213 Second Decision Subunit 1214 Deduplication Subunit 1215 Expansion Subunit 1221 First predicted subunit 1222 First Coordination Sub-Unit 1223 Third Decision Subunit 1224 subunits 1225 Second predicted subunit 1226 Second Coordination Sub-Unit 1227 Fourth Decision Subunit 1741 Analysis Subunit 1742 subunits 1743 subunits 1744 First Decision Subunit

Claims

1. 1. An image processing method, comprising: a step of acquiring a detection target image, performing image detection of a critical portion of the detection target image, and acquiring a set of candidate boxes, the set of candidate boxes including candidate boxes for marking an image detection result; a step of obtaining a set of target boxes by performing a process of eliminating duplicates of candidate boxes in the set of candidate boxes based on the detection target image, obtaining adjustment boxes by correcting positions of the candidate boxes in the set of target boxes, and generating a set of adjustment boxes and a set of clipped images corresponding to the set of adjustment boxes based on the adjustment boxes, wherein the set of clipped images includes N clipped images (N is a positive integer), and one clipped image is obtained by clipping the detection target image based on one adjustment box in the set of adjustment boxes; performing a recognition and segmentation process on each of the N clipped images to obtain a segmentation result associated with the critical portion, the segmentation result including M (M is a positive integer equal to or less than N) segmented images; and performing a correction process on the adjustment boxes corresponding to the M divided images in the adjustment box set based on the coordinate positions of the M divided images, thereby obtaining M correction boxes for indicating the critical portion in the detection target image. An image processing method comprising:

2. before the step of obtaining a set of candidate boxes by image detecting a critical portion of the detection target image, obtaining a first sample image for training a first detection model and a first sample label for indicating an actual classification of the first sample image, the first sample image being obtained by preprocessing a critical portion of an original sample image; Invoking the first detection model and detecting the first sample image to obtain a predicted classification confidence of the first sample image for the critical region; and training the first detection model based on the predicted classification confidence of the first sample image and the actual classification of the first sample image to obtain an image detection model for image detection of the critical portion of the detection target image; The step of acquiring a detection target image and detecting a critical portion of the detection target image to acquire a set of candidate boxes includes: a step of acquiring a detection target image, and performing image detection of a critical portion of the detection target image using the image detection model to acquire a set of candidate boxes; 2. The image processing method according to claim 1.

3. The step of obtaining an image detection model for image detection of a critical portion of the detection target image by training the first detection model based on the predicted classification confidence of the first sample image and the actual classification of the first sample image includes: determining a model loss of the first detection model by performing a loss calculation process between the predicted classification confidence of the first sample image and the actual classification of the first sample image; obtaining a first model training result by training the first detection model based on a model loss of the first detection model; If the first model training result indicates that the trained first detection model satisfies a first model convergence condition, selecting the first detection model that satisfies the first model convergence condition as a second detection model; performing a structural analysis of the network structure of the second detection model to obtain an analysis result, and generating an image detection model for image detection of a critical portion of the detection target image based on the analysis result and the second detection model; 3. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

4. the step of acquiring an analysis result by performing a structural analysis of a network structure of the second detection model, and generating an image detection model for image detection of a critical portion of the detection target image based on the analysis result and the second detection model, performing a structural analysis of the network structure of the second detection model to obtain an analysis result; If the analysis result indicates that a fully connected layer exists in the network structure of the second detection model, replacing the fully connected layer with a first convolution layer in which a sliding step size is configured; Obtaining a second convolutional layer by assigning values ​​to the first convolutional layer based on network parameters of the fully connected layer; determining a second detection model including the second convolution layer as an image detection model for image detection of a critical portion of the detection target image; 4. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

5. The step of obtaining a set of target boxes by performing a process of eliminating duplicates of the candidate boxes in the set of candidate boxes based on the detection target image, obtaining adjustment boxes by correcting the positions of the candidate boxes in the set of target boxes, and generating a set of adjustment boxes and a set of cutout images corresponding to the set of adjustment boxes based on the adjustment boxes includes: A step of obtaining a set of target boxes by performing a process of de-duplication of candidate boxes in the set of candidate boxes based on a shape adaptive local maximum suppression de-duplication rule, the set of target boxes including candidate boxes X ｉ where i is a positive integer less than or equal to H, and H denotes the total number of candidate boxes in the set of boxes to be processed; Invoke the image shift adjustment model and calculate the candidate box X by the target image. ｉ By correcting the position of the adjustment box Y ｉ and the adjustment box Y ｉ and obtaining a cropped image corresponding to When the H adjustment boxes are acquired, performing a deduplication process on the H adjustment boxes based on the deduplication rule; determining the adjustment boxes after the deduplication process as an adjustment box set, and determining the clipped images corresponding to the adjustment boxes after the deduplication process as a clipped image set corresponding to the adjustment box set, 5. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

6. The candidate box set is obtained by invoking an image detection model to detect critical regions of the target image, the candidate box set including K candidate boxes (K is a positive integer), and the image detection model is further used to determine a predicted classification confidence for each of the K candidate boxes; The step of obtaining a set of boxes to be processed by performing a process of de-duplication of candidate boxes in the set of candidate boxes based on a shape adaptive maximum suppression de-duplication rule includes: obtaining K clipping boxes by performing a clipping process on each of the K candidate boxes according to an image clipping ratio set by a shape adaptive maximum suppression overlap elimination rule; performing a sorting process on the K clipping boxes according to the K predicted classification confidences to obtain a sorting result; determining a clipping box having a maximum predicted classification reliability in the sorted result as a first clipping box, and determining (K-1) clipping boxes other than the first clipping box in the sorted result as a filtering target set; obtaining a set of reserved boxes by performing a duplicate elimination process on the K clipping boxes based on an overlapping degree between the first clipping box and each clipping box in the filtering target set; and performing image expansion processing on each clipping box in the set of reserved boxes based on the image clipping rate to obtain a set of boxes to be processed.

6. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

7. The step of acquiring a set of reserved boxes by performing a process of eliminating duplicates of the K clipping boxes based on an overlapping degree between the first clipping box and each clipping box in the filtering target set includes: Retaining the first clipping box and determining an overlap between the first clipping box and each clipping box in the filtered set; If there is an overlapping clipping box in the filtering target set whose overlap degree is greater than an overlap degree threshold, filtering the overlapping clipping box in the filtering target set; a step of determining a clipping box having a maximum predicted classification reliability in the filtering target set after filtering as a second clipping box, and determining clipping boxes other than the second clipping box in the filtering target set after filtering as a new filtering target set; reserving the second clipping box, and continuing a deduplication process on the new filtering target set based on the degree of overlap between the second clipping box and each clipping box in the new filtering target set until the filtering target set after the deduplication process is empty, and determining the reserved clipping box as a reserved box set, wherein the reserved box set includes the first clipping box and the second clipping box.

7. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

8. Invoke the image shift adjustment model and calculate the candidate box X by the target image. ｉ By correcting the position of the adjustment box Y ｉ and the adjustment box Y ｉ and a corresponding cropped image, The image shift adjustment model is called, and the candidate box X is calculated by the detection target image. ｉ A step of obtaining first regression parameters by predicting the deviation amount of Based on the first regression parameters, the candidate box X ｉ By correcting the position of the candidate box X ｉ obtaining a first adjustment box corresponding to If the first regression parameter falls within a regression parameter threshold range, the candidate box X ｉ The first adjustment box corresponding to the candidate box X ｉ Adjustment box Y corresponding to ｉ and determining The adjustment box Y ｉ The adjustment box Y ｉ and obtaining a cropped image corresponding to 8. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

9. The candidate box X ｉ After obtaining the first adjustment box corresponding to If the first regression parameter does not belong to the regression parameter threshold range, the image shift adjustment model is invoked, and the candidate box X ｉ obtaining second regression parameters by predicting the deviation of the first adjustment box corresponding to Based on the second regression parameters, the candidate box X ｉ By correcting the position of the first adjustment box corresponding to the candidate box X ｉ obtaining a second adjustment box corresponding to If the second regression parameter falls within the regression parameter threshold range, the candidate box X ｉ The second adjustment box corresponding to the candidate box X ｉ Adjustment box Y corresponding to ｉ and determining the 9. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

10. Before calling the image shift adjustment model, obtaining a second sample image for training an initial shift adjustment model and a second sample label for indicating an actual coordinate position where a critical portion of the second sample image is located; calling the initial shift adjustment model and performing shift prediction processing for the second sample image to obtain predicted regression parameters for the second sample image; obtaining a predicted coordinate position of the second sample image by performing position correction of the coordinate position of the second sample image based on the predicted regression parameters; training the initial shift adjustment model based on the predicted coordinate position and the actual coordinate position to obtain a second model training result; If the second model training result indicates that the initial shift adjustment model after training satisfies the second model convergence condition, determining the initial shift adjustment model that satisfies the second model convergence condition as the image shift adjustment model.

10. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

11. The step of obtaining a segmentation result associated with the critical portion by performing a recognition and segmentation process on each of the N number of clipped images, determining a processing target image from the N cutout images; calling an image recognition division model, and recognizing and processing the target image according to the image classification branch in the image recognition division model, thereby obtaining a predicted classification confidence corresponding to the target image; a step of removing non-critical portions of the processing target image by image division branching in the image recognition division model, thereby obtaining a divided image corresponding to the processing target image; and determining a segmented image corresponding to the processing target image as a segmentation result associated with the critical region when the predicted classification reliability corresponding to the processing target image is greater than a reliability threshold.

11. The image processing method according to claim 1.

12. Before calling the image recognition segmentation model, obtaining sample data for training an initial recognition segmentation model and sample labels corresponding to the sample data; Obtaining a first recognition segmentation model by training a public network layer and an image segmentation branch in the initial recognition segmentation model based on a third sample image in the sample data and a third sample label for indicating an actual mask image of the third sample image; Obtaining a second recognition segmentation model by training a public network layer and an image classification branch in the first recognition model according to a fourth sample image in the sample data and a fourth sample label for indicating an actual classification of the fourth sample image; Obtaining a third recognition segmentation model by training a public network layer and an image segmentation branch in the second recognition segmentation model based on the third sample image and the third sample label; and further comprising: locking a public network layer and an image segmentation branch in the third recognition segmentation model; and training an image classification branch in the locked third recognition segmentation model based on the fourth sample image and the fourth sample label, thereby obtaining an image recognition segmentation model.

12. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

13. The step of obtaining a third recognition segmentation model by training a public network layer and an image segmentation branch in the second recognition segmentation model based on the third sample image and the third sample label includes: Invoking the second recognition segmentation model to remove non-critical portions of the third sample image and process the third sample image to obtain a predicted mask image corresponding to the third sample image; scanning pixels of the third sample image and determining the scanned pixels as pixels to be processed; determining a reliability that the pixel to be processed is in the actual mask image indicated by the third sample label as a first reliability, and determining a reliability that the pixel to be processed is in the predicted mask image as a second reliability; determining a model loss of the second recognition segmentation model by performing a loss calculation process between the first reliability and the second reliability; Obtaining a third model training result by training a public network layer and an image segmentation branch in the second recognition segmentation model based on the model loss of the second recognition segmentation model; If the third model training result indicates that the second recognition division model after training satisfies a first branching convergence condition in the third model convergence condition, determining the second recognition division model that satisfies the first branching convergence condition as the third recognition division model.

13. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

14. The step of obtaining an image recognition segmentation model by locking a public network layer and an image segmentation branch in the third recognition segmentation model, and training an image classification branch in the locked third recognition segmentation model based on the fourth sample image and the fourth sample label, includes: locking a public network layer and an image division branch in the third recognition division model, and determining the locked third recognition division model as a fourth recognition division model; calling the fourth recognition segmentation model and performing recognition processing on the fourth sample image to determine a sample prediction reliability corresponding to the fourth sample image; determining a model loss of the fourth recognition segmentation model by performing a loss calculation process between the actual classification indicated by the fourth label and the sample prediction reliability; obtaining a fourth model training result by training an image classification branch in the fourth recognition division model based on the model loss of the fourth recognition division model; If the fourth model training result indicates that the trained fourth recognition division model satisfies a second branching convergence condition in the third model convergence condition, determining the fourth recognition division model that satisfies the second branching convergence condition as the image recognition division model.

14. The image processing method according to claim 1, wherein the first and second inputs are input to the image processing unit.

15. An image processing device, an image detection module that, upon acquiring a detection target image, performs image detection of a critical portion of the detection target image to acquire a set of candidate boxes, the set of candidate boxes including candidate boxes for marking an image detection result; a deduplication adjustment module that acquires a processing target box set by performing deduplication processing on candidate boxes within the candidate box set based on the detection target image, acquires adjustment boxes by correcting positions of the candidate boxes within the processing target box set, and generates an adjustment box set and a clipped image set corresponding to the adjustment box set based on the adjustment boxes, wherein the clipped image set includes N clipped images (N is a positive integer), and one clipped image is obtained by clipping the detection target image based on one adjustment box within the adjustment box set; a recognition and segmentation module that performs a recognition and segmentation process on each of the N clipped images to obtain a segmentation result associated with the critical portion, the segmentation result including M (M is a positive integer equal to or less than N) segmented images; a correction module that performs a correction process on the adjustment boxes corresponding to the M divided images in the adjustment box set based on the coordinate positions of the M divided images, thereby obtaining M correction boxes that indicate the critical portion in the detection target image, 1. An image processing device comprising:

16. A computer device comprising a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, the network interface providing a data communication function, the memory storing a computer program, and the processor calling the computer program to cause the computer device to execute the image processing method according to any one of claims 1 to 14.

1. A computer device characterized by:

17. 15. A computer-readable storage medium having a computer program stored thereon, the computer program being configured to cause a computer device having the processor to execute the image processing method according to any one of claims 1 to 14 when loaded and executed by a processor.

18. 15. A computer program product comprising a computer program, the computer program being stored on a computer-readable storage medium, the computer program being configured to, when read and executed by a processor, cause a computing device comprising the processor to perform the image processing method of any one of claims 1 to 14.

Images