Image processing method and device, computer readable storage medium and electronic equipment
By using image key points in a multimodal big model to segment object areas and generate annotation information, the problem of image description deviating from the topic is solved, and the accuracy and efficiency of description are improved.
Patent Information
- Application Number
- CN202411587912.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-02-28
AI Technical Summary
Multimodal big models tend to deviate from the image subject when processing images.
By obtaining key points in the image, segmenting out object areas, and inputting these areas into the trained image processing model, outputting corresponding annotation information.
Improves the accuracy and robustness of image segmentation, reduces computational complexity and memory consumption, enhances the accuracy of annotations, and makes the description more suitable for the image theme.
Smart Images
Figure CN119540954A_ABST
Abstract
Description
Background Art
[0002] A multimodal large model refers to a large-scale machine learning model that can process and integrate multiple different types of data.
[0003] Based on the multimodal large model, the input image can be understood, and the image caption can be generated or the image can be described in language. However, the multimodal large model sometimes produces captions or descriptions that deviate from the image theme.
[0004] In view of this, there is an urgent need in the art to develop a new image processing method and device.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure. Summary of the invention
[0006] The purpose of the present disclosure is to provide an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby overcoming the technical problem that subtitles or descriptions deviate from the image subject due to the limitations of related technologies at least to a certain extent.
[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0008] According to a first aspect of the present disclosure, there is provided an image processing method, comprising:
[0009] Get image key points in the input image;
[0010] Segmenting object regions from the input image based on the image key points; each of the object regions contains at least one of the image key points;
[0011] The object area is input into a trained image processing model, and annotation information corresponding to the input image is output through the image processing model.
[0012] In an exemplary embodiment of the present disclosure, inputting the object area into a trained image processing model, and outputting annotation information corresponding to the input image through the image processing model, includes:
[0013] Filtering out a main object region from the object region;
[0014] The main object area is input into the image processing model, and annotation information corresponding to the input image is output through the image processing model.
[0015] In an exemplary embodiment of the present disclosure, filtering out a main object region from the object region includes:
[0016] Obtaining the pixel quantity corresponding to each of the object regions;
[0017] Screening out a candidate subject object region from the plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold;
[0018] The subject object region is screened out from the candidate subject object regions.
[0019] In an exemplary embodiment of the present disclosure, the step of selecting a candidate subject object region from a plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold comprises:
[0020] In response to the pixel quantity being greater than or equal to the preset pixel quantity threshold, determining the current object region as the candidate subject object region;
[0021] In response to the pixel quantity being less than the preset pixel quantity threshold, it is determined that the current object region is not the candidate main object region, and the current object region is discarded.
[0022] In an exemplary embodiment of the present disclosure, the step of filtering out the subject object region from the candidate subject object regions includes:
[0023] Obtaining the object type contained in each candidate subject object region;
[0024] Sorting the plurality of candidate subject object regions according to the priority sorting results corresponding to the plurality of pre-configured object types to obtain a sorting sequence;
[0025] The main object area is determined according to the sorting sequence.
[0026] In an exemplary embodiment of the present disclosure, determining the subject object area according to the sorting sequence includes:
[0027] The candidate main object region ranked first in the sorting sequence is determined as the main object region.
[0028] In an exemplary embodiment of the present disclosure, the image key point includes an intersection point of the image center point and a preset ratio dividing line.
[0029] According to a second aspect of the present disclosure, there is provided an image processing apparatus, comprising:
[0030] A key point acquisition module, used to acquire image key points in an input image;
[0031] An object segmentation module, used to segment object regions from the input image based on the image key points; each of the object regions contains at least one of the image key points;
[0032] The annotation generation module is used to input the object area into a trained image processing model, and output annotation information corresponding to the input image through the image processing model.
[0033] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method described in the first aspect is implemented.
[0034] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image processing method described in the first aspect above by executing the executable instructions.
[0035] It can be seen from the above technical solutions that the image processing method, image processing device, computer-readable storage medium and electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:
[0036] In the technical solutions provided in some embodiments of the present disclosure, image key points in an input image are obtained, and object regions are segmented from the input image based on the image key points; each object region contains at least one image key point, and the object region is input into a trained image processing model, and the image processing model outputs annotation information corresponding to the input image. On the one hand, the key point-guided segmentation method can more accurately locate and segment objects in the image, which helps the model to better understand the boundaries and shapes of objects and improve the accuracy and robustness of segmentation; further, after the image is segmented into multiple object regions, the model only needs to process these local regions instead of the entire image, thereby reducing computational complexity and memory consumption and improving processing speed; on the other hand, given that the image key points have a certain correlation with the location of the image subject, the model can also focus on the image subject when understanding the image, and output annotation information that is more in line with the image subject, thereby improving annotation accuracy.
[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0039] Figure 1 A framework diagram showing an image processing method in related art;
[0040] Figure 2 A schematic diagram showing a flow chart of an image processing method in an embodiment of the present disclosure;
[0041] Figure 3 A schematic diagram showing key points of an image in an embodiment of the present disclosure;
[0042] Figure 4 A schematic diagram showing the center point of an exemplary image in the present disclosure;
[0043] Figure 5 A schematic diagram showing image thirds in an embodiment of the present disclosure;
[0044] Figure 6-Figure 9 A schematic diagram showing multiple object regions segmented from an input image in an embodiment of the present disclosure;
[0045] Fig.10 A schematic diagram showing a flow chart of how to input an object region into a trained image processing model and output annotation information corresponding to the input image through the image processing model in an embodiment of the present disclosure;
[0046] Fig.11 A schematic diagram showing a flow chart of how to filter out a subject object area from an object area in an embodiment of the present disclosure;
[0047] Fig.12 A schematic diagram showing a flow chart of how to determine a subject object area in an embodiment of the present disclosure;
[0048] Fig.13 The overall architecture diagram of the image processing method in the embodiment of the present disclosure is shown;
[0049] Fig.14 A schematic diagram showing the overall flow of the image processing method in an embodiment of the present disclosure is shown;
[0050] Fig.15 A schematic diagram showing the structure of an image processing device in an exemplary embodiment of the present disclosure is shown;
[0051] Fig.16 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0053] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.
[0054] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.
[0055] Today's large multimodal models can understand the input images, generate image captions, or provide detailed language descriptions of the images.
[0056] refer to Figure 1 , Figure 1 The framework diagram of the image processing method in the related art is shown, including steps S101 to S103:
[0057] In step S101, an image is input;
[0058] In step S102, the multimodal large model is called;
[0059] In step S103, a picture caption or text description of the image is output.
[0060] Due to the fact that a large number of training images and texts are collected from the Internet, although the relevance of text and images can be guaranteed through technical screening, it is impossible to ensure that the main content of the image is the main description of the text. For example, the most appropriate description is "a man in blue clothes is walking on a zebra crossing with a dog.", but the text gives "the traffic light on the road is green."
[0061] Due to the problem of training samples, the large model sometimes cannot grasp the main content of the image, identify the "protagonist" in the image, and thus cannot output descriptive information related to the main content. The subtitles or descriptions generated for the image are not detailed or accurate, and deviate from the main content of the image.
[0062] Based on the key points of the image, the present invention first extracts the object area in the image that is most likely to be the subject, and inputs the object area as a prompt (for example, "What is the man in blue clothes in the image doing?") into the multimodal large model, so that the large model has a more accurate subject when describing the image. The multimodal large model can generate a complete description with the image subject as the subject, and form a more accurate image caption, so that in related scenarios such as smart album photo classification, the classification of photos and the generation of various tags can be optimized.
[0063] In an embodiment of the present disclosure, firstly, an image processing method is provided, which at least to some extent overcomes the defect in the related art that the subtitles or descriptions deviate from the image subject.
[0064] Figure 2 A schematic flow chart of an image processing method in an embodiment of the present disclosure is shown. The execution subject of the image processing method may be a server that processes the image.
[0065] refer to Figure 2 According to an embodiment of the present disclosure, an image processing method comprises the following steps:
[0066] Step S210, obtaining image key points in the input image;
[0067] Step S220, segmenting object regions from the input image based on the image key points; each object region contains at least one image key point;
[0068] Step S230, input the object area into the trained image processing model, and output annotation information corresponding to the input image through the image processing model.
[0069] exist Figure 2In the technical solution provided by the illustrated embodiment, by acquiring image key points in an input image, object regions are segmented from the input image based on the image key points; each object region contains at least one image key point, and the object region is input into a trained image processing model, and the image processing model outputs annotation information corresponding to the input image. On the one hand, the key point-guided segmentation method can more accurately locate and segment objects in the image, which helps the model to better understand the boundaries and shapes of objects and improve the accuracy and robustness of segmentation; further, after the image is segmented into multiple object regions, the model only needs to process these local regions instead of the entire image, thereby reducing computational complexity and memory consumption and improving processing speed; on the other hand, given that the image key points have a certain correlation with the location of the image subject, the model can also focus on the image subject when understanding the image, and output annotation information that is more in line with the image subject, thereby improving annotation accuracy.
[0070] The following Figure 2 The specific implementation process of each step is described in detail:
[0071] In step S210, image key points in the input image are obtained.
[0072] In this step, the key points of the image in the input image can be obtained. Exemplarily, the key points of the image can include the intersection of the center point of the image and the preset ratio dividing line. Figure 3 , Figure 3 A schematic diagram showing key points of an image in an embodiment of the present disclosure is shown, Figure 3 The intersection point of the center is the center point of the above image, and the other four intersection points are the intersection points of the above preset ratio dividing lines.
[0073] The center point of the image may be the geometric center position of the image, for example, the center point in a two-dimensional coordinate system. For example, for an image with a width of W and a height of H, the coordinates of the center point may be calculated by the following formula: the horizontal coordinate xc of the center point = W / 2, the vertical coordinate yc of the center point = H / 2. For example, if the size of an image is 800x600 pixels, then its center point is (400,300). For example, refer to Figure 4 , Figure 4 A schematic diagram showing the center point of an exemplary image in the present disclosure.
[0074] Optionally, the image center point may also be the center point of the image content, such as the focus of the image content or the position of the main object, which is not a fixed mathematical position but is determined according to the actual content of the image. For example, when taking a portrait photo, the position of the face may be considered as the "center point" of the image.
[0075] The intersection points of the preset ratio dividing lines in the above image may be points formed by the intersection of these dividing lines when the image is divided into multiple areas according to a certain ratio.
[0076] The above-mentioned preset ratio dividing lines refer to lines that divide the image horizontally or vertically into several areas according to a predetermined ratio (such as 1:1, 2:1, 3:2, etc.). For example, an image is horizontally divided into two halves at a ratio of 1:1, or vertically divided into three parts at a ratio of 2:1. Both can be set according to actual conditions, and the present disclosure does not make any special limitations on this.
[0077] Exemplarily, the above-mentioned preset ratio dividing line can be a third line of the image. Exemplarily, refer to Figure 5 , Figure 5 A schematic diagram showing image thirds in an embodiment of the present disclosure.
[0078] Thus, the above intersection points can be the third points of the image. The "Rule of Thirds" is a commonly used design principle. According to this principle, the image is divided into nine equal parts by drawing two equally spaced vertical lines and two equally spaced horizontal lines on the image. These four lines will form four intersection points in the image, and these four intersection points are called "intersection points". Specifically: the width of the image will be divided into three equal parts, forming two vertical lines, and the height of the image will also be divided into three equal parts, forming two horizontal lines.
[0079] By determining the image key points from the input image, it is convenient to quickly and accurately locate the key objects in the image based on the image key points. This ensures that the input of the image processing model contains fewer non-key objects to avoid interference in the model's understanding of the image, while reducing the amount of information that the model needs to process when understanding the image.
[0080] Next reference Figure 2 In step S220, object regions are segmented from the input image based on image key points; each object region contains at least one image key point.
[0081] In this step, object regions can be segmented from the input image based on the above image key points (each object region can contain at least one image key point). Specifically, multiple object regions can be segmented from the above input image based on the above image key points through key point prompt+SAM or SAM2. Among them, SAM (SegmentAnything Model) is a new image segmentation model developed by Meta AI (formerly Facebook AI Research). It aims to solve a core challenge in the field of image segmentation: how to enable machine learning models to flexibly segment any type of objects, not just those specific categories that have appeared in the training data. The design concept of SAM is to enable the model to generalize to unseen object types and achieve the goal of "segmenting anything".
[0082] For example, please refer to Figure 6-Figure 9 , Figure 6-Figure 9 A schematic diagram showing multiple object regions segmented from an input image in an embodiment of the present disclosure, each of which contains at least one image key point (i.e. Figure 6-Figure 9 black dot in the middle).
[0083] Next reference Figure 2 In step S230, the object area is input into the trained image processing model, and the image processing model outputs annotation information corresponding to the input image.
[0084] In this step, in an optional implementation, the segmented object region may be input into a trained image processing model, and the image processing model may perform image understanding processing based on the object region to output annotation information corresponding to the input image.
[0085] The “annotation information corresponding to the input image” refers to the process of describing or marking an image, that is, explaining or annotating the content in the image with text or other forms of information. This information may include the category, location, attributes and other relevant information of the target in the image.
[0086] For example, annotation information can involve the following aspects:
[0087] Target category: the type of object or entity contained in the image, such as a person, car, tree, etc.
[0088] Location information: The location of each object in the image, usually marked by a bounding box;
[0089] Detailed description: A detailed text description of the image content, including the target's actions, status, and relationships;
[0090] Attribute information: color, size, posture and other characteristics of the target;
[0091] Scene information: describes the contextual information of the entire image scene, such as weather, environment, etc.
[0092] In another optional embodiment, reference may be made to Fig.10 , Fig.10 A flow chart showing how to input an object region into a trained image processing model and output annotation information corresponding to the input image through the image processing model in an embodiment of the present disclosure includes steps S1001 and S1002:
[0093] In step S1001, a main object region is screened out from the object region.
[0094] In this step, the main object area can be screened out from the object area. Fig.11 , Fig.11 A flowchart showing how to filter out a main object area from an object area in an embodiment of the present disclosure includes steps S1101 to S1103:
[0095] In step S1101, the number of pixels corresponding to each object area is obtained.
[0096] In this step, the number of pixels corresponding to each object area can be obtained, where a pixel is the smallest unit of an image, and each pixel has a specific color value. For a grayscale image, a pixel value is usually an integer between 0 and 255; for a color image, each pixel is usually composed of three components: red, green, and blue, and each component is also an integer between 0 and 255. The number of pixels refers to the number of pixels contained in a specific area in an image.
[0097] In step S1102, a candidate main object region is screened out from a plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold.
[0098] In this step, candidate main object areas can be screened out from multiple object areas based on the comparison results between the pixel quantity and the preset pixel quantity threshold. For example, a maximum pixel quantity can be preset first, and then a preset multiple of the maximum pixel quantity (for example: 0.5 times) can be set as the above-mentioned preset pixel quantity threshold.
[0099] Thus, when the pixel quantity of the current object area is greater than or equal to the above-mentioned preset pixel quantity threshold, it can be determined that the current object area is a candidate subject object area, and when the pixel quantity of the current object area is less than the above-mentioned preset pixel quantity threshold, it can be determined that the current object area is not the above-mentioned candidate subject object area, and the above-mentioned current object area is discarded.
[0100] After the candidate main object regions are screened out, reference may be made to step S1103 to screen out the main object region from the candidate main object regions.
[0101] In this step, the object type contained in each candidate subject object area can be obtained. The object type refers to the category or label to which the object belongs. For example, these categories can be specific object names, such as "person", "car", "tree", etc.
[0102] Afterwards, the plurality of candidate subject object regions may be sorted according to the priority sorting results corresponding to the plurality of pre-configured object types to obtain a sorting sequence, and then the candidate subject object region ranked first in the sorting sequence may be determined as the subject object region.
[0103] Exemplarily, the priority ranking results corresponding to the above-mentioned multiple object types may be: people>animals>landscapes>plants>objects, and may also be set or adjusted according to actual conditions, and the present disclosure does not make any special limitation on this.
[0104] It should be noted that, if there is a parallel object at the first position in the sorting sequence, the one with a larger pixel amount can be preferentially selected as the main object area.
[0105] refer to Fig.12 , Fig.12 A flowchart showing how to determine the subject object area in the embodiment of the present disclosure includes steps S1201 to S1205:
[0106] In step S1201, start;
[0107] In step S1202, an object region is segmented from the input image based on the image key points;
[0108] In step S1203, it is determined one by one whether the pixel quantity of the object area is greater than or equal to a preset pixel quantity threshold, and those less than the threshold are discarded, and those greater than the threshold are determined as candidates;
[0109] In step S1204, a subject object region is determined from candidate subject object regions according to the priority of the object type;
[0110] In step S1205, end.
[0111] After filtering out the main object area, you can then refer to Fig.10 In step S1002, the main object area is input into the image processing model, and the annotation information corresponding to the input image is output through the image processing model.
[0112] In this step, the main object area may be input into the image processing model, and the annotation information corresponding to the input image may be output through the image processing model.
[0113] The present invention inputs the above-mentioned main object area into the image processing model, and outputs the annotation information corresponding to the input image through the image processing model, thereby providing a subject describing the image for the multimodal large model, so that the large model can better generate image descriptions.
[0114] refer to Fig.13 , Fig.13 The overall architecture diagram of the image processing method in the embodiment of the present disclosure is shown, including steps S1301 to S1305:
[0115] In step S1301, an image is input;
[0116] In step S1302, the object area on the image is extracted according to the image key points;
[0117] In step S1303, a main object area is selected;
[0118] In step S1304, the multimodal image understanding large model is called;
[0119] In step S1305, the image caption or text description is output.
[0120] refer to Fig.14 , Fig.14 The overall flow chart of the image processing method in the embodiment of the present disclosure is shown, including steps S1401 to S1404:
[0121] In step S1401, an image is input;
[0122] In step S1402, object regions are extracted from the input image according to the image key points to obtain multiple object regions;
[0123] In step S1403, a subject object is selected from a plurality of object areas;
[0124] In step S1404, the trained image processing model is used to output annotation information corresponding to the input image.
[0125] Based on the above technical solutions, the present disclosure has at least the following technical effects:
[0126] First, by extracting the object area in the image according to the image key points, the object area is input into the multimodal large model as a prompt, so that the key objects in the image can be quickly and accurately located according to the image key points in the future, thereby ensuring that there are fewer non-key objects in the input of the image processing model to avoid interference in the model's understanding of the image;
[0127] Second, by screening the main object area from the object area and inputting the main object area into the multimodal large model as a prompt, the multimodal large model can be provided with a subject to describe the image, so that the large model can better generate image descriptions.
[0128] The present disclosure also provides an image processing device, Fig.15 FIG. 4 is a schematic diagram showing the structure of an image processing device in an exemplary embodiment of the present disclosure; Fig.15 As shown, the image processing apparatus 1500 may include a module 1510, a module 1520 and a module 1530. Among them:
[0129] A key point acquisition module 1510 is used to acquire image key points in an input image;
[0130] An object segmentation module 1520 is used to segment object regions from the input image based on the image key points; each of the object regions contains at least one of the image key points;
[0131] The annotation generation module 1530 is used to input the object area into a trained image processing model, and output annotation information corresponding to the input image through the image processing model.
[0132] In an exemplary embodiment of the present disclosure, the annotation generation module 1530 inputs the object area into a trained image processing model, and outputs annotation information corresponding to the input image through the image processing model, including:
[0133] Filtering out a main object region from the object region;
[0134] The main object area is input into the image processing model, and annotation information corresponding to the input image is output through the image processing model.
[0135] In an exemplary embodiment of the present disclosure, the annotation generating module 1530 filters out a main object region from the object region, including:
[0136] Obtaining the pixel quantity corresponding to each of the object regions;
[0137] Screening out a candidate subject object region from the plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold;
[0138] The subject object region is screened out from the candidate subject object regions.
[0139] In an exemplary embodiment of the present disclosure, the annotation generating module 1530 selects a candidate subject object region from the plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold, including:
[0140] In response to the pixel quantity being greater than or equal to the preset pixel quantity threshold, determining the current object region as the candidate subject object region;
[0141] In response to the pixel quantity being less than the preset pixel quantity threshold, it is determined that the current object region is not the candidate main object region, and the current object region is discarded.
[0142] In an exemplary embodiment of the present disclosure, the annotation generating module 1530 selects the subject object region from the candidate subject object regions, including:
[0143] Obtaining the object type contained in each candidate subject object region;
[0144] Sorting the plurality of candidate subject object regions according to the priority sorting results corresponding to the plurality of pre-configured object types to obtain a sorting sequence;
[0145] The main object area is determined according to the sorting sequence.
[0146] In an exemplary embodiment of the present disclosure, the annotation generating module 1530 determines the subject object area according to the sorting sequence, including:
[0147] The candidate main object region ranked first in the sorting sequence is determined as the main object region.
[0148] In an exemplary embodiment of the present disclosure, the image key point includes an intersection point of the image center point and a preset ratio dividing line.
[0149] The specific details of each module in the above-mentioned image processing device have been described in detail in the corresponding image processing method, so they will not be repeated here.
[0150] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0151] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0152] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0153] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.
[0154] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0155] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0156] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.
[0157] In addition, an electronic device capable of implementing the above method is also provided in an embodiment of the present disclosure.
[0158] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0159] Refer to the following Fig.16 1600 according to this embodiment of the present disclosure is described. Fig.16 The electronic device 1600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0160] like Fig.16 As shown, the electronic device 1600 is in the form of a general computing device. The components of the electronic device 1600 may include, but are not limited to: at least one processor 1610, at least one memory 1620, a bus 1630 connecting different system components (including the memory 1620 and the processor 1610), and a display 1640.
[0161] The memory stores program codes, which can be executed by the processor 1610, so that the processor 1610 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processor 1610 can execute the following steps: Figure 2 As shown in: Step S210, obtaining image key points in the input image; Step S220, segmenting object regions from the input image based on the image key points; each object region contains at least one image key point; Step S230, inputting the object region into the trained image processing model, and outputting annotation information corresponding to the input image through the image processing model.
[0162] The memory 1620 may include a readable medium in the form of a volatile storage, such as a random access memory (RAM) 16201 and / or a cache memory 16202 , and may further include a read-only memory (ROM) 16203 .
[0163] Memory 1620 may also include a program / utility 16204 having a set (at least one) of program modules 16205, such program modules 16205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0164] Bus 1630 may be a bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.
[0165] The electronic device 1600 may also communicate with one or more external devices 1700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1600, and / or communicate with any device that enables the electronic device 1600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1650. Furthermore, the electronic device 1600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1660. As shown, the network adapter 1660 communicates with other modules of the electronic device 1600 via a bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0166] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
Claims
1. An image processing method, characterized in that: include: Get image key points in the input image; Segmenting an object region from the input image based on the image key points; Each of the object regions contains at least one of the image key points; The object area is input into a trained image processing model, and annotation information corresponding to the input image is output through the image processing model.
2. The method according to claim 1, characterized in that The step of inputting the object region into a trained image processing model and outputting annotation information corresponding to the input image through the image processing model includes: Filtering out a main object region from the object region; The main object area is input into the image processing model, and annotation information corresponding to the input image is output through the image processing model.
3. The method according to claim 2, characterized in that The step of filtering out the main object region from the object region comprises: Obtaining the pixel quantity corresponding to each of the object regions; Screening out a candidate subject object region from the plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold; The subject object region is screened out from the candidate subject object regions.
4. The method according to claim 3, characterized in that The step of selecting a candidate subject object region from the plurality of object regions according to a comparison result between the pixel quantity and a preset pixel quantity threshold comprises: In response to the pixel quantity being greater than or equal to the preset pixel quantity threshold, determining the current object region as the candidate subject object region; In response to the pixel quantity being less than the preset pixel quantity threshold, it is determined that the current object region is not the candidate main object region, and the current object region is discarded.
5. The method according to claim 3, characterized in that: The step of selecting the subject object region from the candidate subject object regions comprises: Obtaining the object type contained in each candidate subject object region; Sorting the plurality of candidate subject object regions according to the priority sorting results corresponding to the plurality of pre-configured object types to obtain a sorting sequence; The main object area is determined according to the sorting sequence.
6. The method according to claim 5, characterized in that The determining the main object area according to the sorting sequence includes: The candidate main object region ranked first in the sorting sequence is determined as the main object region.
7. The method according to any one of claims 1 to 6, characterized in that: The key points of the image include the intersection of the center point of the image and a preset ratio dividing line.
8. An image processing device, characterized in that: include: A key point acquisition module, used to acquire image key points in an input image; An object segmentation module, used to segment object regions from the input image based on the image key points; each of the object regions contains at least one of the image key points; The annotation generation module is used to input the object area into a trained image processing model, and output annotation information corresponding to the input image through the image processing model.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.
10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the image processing method according to any one of claims 1 to 7 by executing the executable instructions.