An image processing method, device, equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose an image processing method, device and equipment and a computer readable storage medium. The method comprises: obtaining a target image, performing feature extraction on the target image, obtaining attribute information of N objects contained in the target image, determining M preselected boxes of the target image according to the attribute information of the N objects, obtaining a candidate position set of the target image according to the M preselected boxes, and determining a target cropping position from the candidate position set according to the attribute information of the N objects; and performing cropping processing on the target image according to the target cropping position to obtain a cropped image. According to the attribute information of the N objects, the target cropping position is determined from the candidate position set, so that the cropped image can include as many key objects (such as objects with a weight higher than a weight threshold) in the N objects as possible and as completely as possible, and the target image can be efficiently processed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an image processing method and device, equipment and computer readable storage medium. BACKGROUND

[0002] With the continuous development of computer technology, a large number of pictures emerge in the network. In the case that the size of the required image is inconsistent with the size of the original image (such as the size of the display image does not match the size of the display area), or the image has a large number of invalid regions (such as black edges), the original image needs to be processed (such as cropping, scaling, etc.). Taking cropping as an example, the commonly used image cropping methods include: cropping the original image according to the region indicated by the current user to obtain the required image; or cropping the original image by a static cropping method (that is, cropping at a fixed cropping position) to obtain the required image. It is found in practice that the method of manually specifying the cropping region is low in efficiency and high in labor cost; and the static cropping method has a fixed cropping position and poor flexibility. SUMMARY

[0003] The embodiments of the present application provide an image processing method, device, equipment and computer readable storage medium, which can efficiently process a target image.

[0004] In one aspect, the embodiments of the present application provide an image processing method, comprising:

[0005] obtaining a target image and performing feature extraction on the target image to obtain attribute information of N objects contained in the target image, N being a positive integer;

[0006] determining M preselected frames of the target image according to the attribute information of the N objects, wherein the number of objects in the jth preselected frame is less than the number of objects in the (j+1)th preselected frame, j and M being positive integers, j being less than M, and M being less than or equal to N;

[0007] obtaining a candidate position set of the target image according to the M preselected frames, and determining a target cropping position from the candidate position set according to the attribute information of the N objects;

[0008] cropping the target image through the target cropping position to obtain a cropped image.

[0009] In one aspect, the embodiments of the present application provide an image processing device, comprising:

[0010] an obtaining unit configured to obtain a target image;

[0011] The processing unit is configured to perform feature extraction on the target image to obtain attribute information of N objects contained in the target image, N being a positive integer; determine M preselected boxes of the target image according to the attribute information of the N objects, wherein the number of objects in the jth preselected box is less than the number of objects in the (j+1)th preselected box, j and M being positive integers, and j being less than M and M being less than or equal to N; obtain a candidate position set of the target image according to the M preselected boxes, and determine a target cropping position from the candidate position set according to the attribute information of the N objects; and perform cropping processing on the target image through the target cropping position to obtain a cropped image.

[0012] In an embodiment, the attribute information of the N objects includes position information of key points corresponding to the N objects respectively; the processing unit is configured to determine the M preselected boxes of the target image according to the attribute information of the N objects, and specifically configured to:

[0013] determine the line distance between the key points of the N objects and the center point of the target image according to the position information of the key points of the N objects, and sort the N objects according to the line distance of each object to obtain an object sorting set;

[0014] wherein the line distance between the key point of the ith object and the center point of the target image is less than or equal to the line distance between the key point of the (i+1)th object and the center point of the target image, i being a positive integer and i being less than N;

[0015] determine the jth preselected box of the target image according to the position information of the first P objects in the object sorting set;

[0016] wherein the jth preselected box includes the first P objects in the image cut by the jth preselected box, and the ratio of the area of the jth preselected box to the area of the target image is greater than a first proportion threshold, P and j being positive integers and P being less than or equal to N and j being less than or equal to M.

[0017] In an embodiment, the processing unit is configured to obtain the candidate position set of the target image according to the M preselected boxes, and specifically configured to:

[0018] obtain a target cropping scale, and adjust the scale of the M preselected boxes based on the target cropping scale to obtain M candidate boxes of the target image;

[0019] slide each candidate box in the target image to obtain the candidate position set of the target image;

[0020] wherein the candidate position corresponding to the target candidate box is the position of the target candidate box in the target image after each sliding.

[0021] In an implementation, the processing unit is configured to adjust the scales of the M preselection boxes based on the target cropping scale to obtain M candidate boxes of the target image, and specifically configured to:

[0022] According to the relationship between the kth preselection box and the N object positions, determine an expansion box corresponding to the kth preselection box, k is a positive integer and k is less than or equal to M;

[0023] According to the distance between the expansion box corresponding to the kth preselection box and the edge of the target image, determine a first inner edge box and a second inner edge box corresponding to the kth preselection box, the area of the first inner edge box is less than the area of the second inner edge box;

[0024] If the area of the target cropping scale is greater than the area of the first inner edge box and less than the area of the second inner edge box, adjust the width-height ratio of the kth preselection box according to the width-height ratio of the target cropping scale to obtain the kth candidate box.

[0025] In an implementation, the image intercepted by the kth preselection box includes Q objects, Q is a positive integer and Q is less than N; the processing unit is configured to determine a first inner edge box and a second inner edge box corresponding to the kth preselection box according to the distance between the kth preselection box and the edge of the target image, and specifically configured to:

[0026] Expand each edge of the kth preselection box to obtain an expansion box corresponding to the kth preselection box;

[0027] The ratio of the area of the image intercepted by the expansion box corresponding to the kth preselection box to the area of the Q+1th object is less than or equal to the second proportion threshold.

[0028] In an implementation, the processing unit is further configured to:

[0029] If the area of the target cropping scale is less than or equal to the area of the first inner edge box, adjust the width-height ratio of the kth preselection box according to the width-height ratio of the first inner edge box, and perform adaptive processing on the kth preselection box according to the target cropping scale to obtain the kth candidate box;

[0030] If the area of the target cropping scale is greater than or equal to the area of the second inner edge box, adjust the width-height ratio of the kth preselection box according to the width-height ratio of the second inner edge box, and perform adaptive processing on the kth preselection box according to the target cropping scale to obtain the kth candidate box;

[0031] The scale of the kth candidate box matches the target cropping scale.

[0032] In an implementation, the attribute information of the N objects includes the weights corresponding to each object and the coordinates of the key points of each object; the processing unit is configured to determine the target cropping position from the candidate position set according to the attribute information of the N objects, and specifically configured to:

[0033] The processing unit is configured to calculate a first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, where x is a positive integer and x is less than or equal to M.

[0034] The processing unit is configured to calculate a second score of the xth candidate position according to the weight of each object in the xth candidate position and the coordinates of the key points of each object.

[0035] The processing unit is configured to calculate a cumulative score of the xth candidate position by summing the first score and the second score.

[0036] The processing unit is configured to determine the candidate position with a cumulative score higher than the score threshold in the candidate position set as the target cropping position.

[0037] In an embodiment, the processing unit is configured to calculate a first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, and specifically configured to:

[0038] The processing unit is configured to calculate a first area score of the xth candidate position according to the area relationship between the first inner frame and the second inner frame corresponding to the preselected frame corresponding to the xth candidate position and the target cropping scale.

[0039] The processing unit is configured to calculate a second area score of the xth candidate position by calculating the ratio of the area of the xth candidate position to the area of the target image.

[0040] The processing unit is configured to calculate a first score of the xth candidate position by performing operation processing on the first area score, the second area score, and the area of the target image.

[0041] In an embodiment, the xth candidate position and at least one object have an overlapping area, and the target object is any one of the at least one object. The processing unit is configured to calculate a second score of the xth candidate position according to the weight of each object in the xth candidate position and the coordinates of the key points of each object, and specifically configured to:

[0042] The processing unit is configured to calculate an object score of the target object according to the overlapping area ratio and the weight of the target object.

[0043] The processing unit is configured to calculate a key point score of the target object according to the coordinates of the key points of the target object.

[0044] The processing unit is configured to calculate a second score of the xth candidate position by performing accumulation operation on the object score and the key point score of the at least one object.

[0045] In an embodiment, the target image belongs to a target video. The processing unit is further configured to:

[0046] If the target image is a first y frames image of the target video, the target image is subjected to edge detection, y is a positive integer;

[0047] In response to the text object existing in the edge region of the target image, a third inner border corresponding to the kth preselected frame is determined, and the text object is included in the third inner border;

[0048] If the area of the target cropping scale is greater than the area of the first inner border and less than the area of the second inner border, the width-height ratio of the kth preselected frame is adjusted according to the width-height ratio of the third inner border, and the kth preselected frame is subjected to adaptive processing according to the target cropping scale, to obtain the kth candidate frame.

[0049] The scale of the kth candidate frame is matched with the target cropping scale.

[0050] Correspondingly, the present application provides an intelligent device, which comprises:

[0051] A processor is configured to load and execute a computer program.

[0052] A computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the image processing method.

[0053] Correspondingly, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is adapted to be loaded and executed by a processor to implement the image processing method.

[0054] Correspondingly, the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the image processing method.

[0055] In the embodiment of the present application, a target image is acquired, and feature extraction is performed on the target image to obtain attribute information of N objects contained in the target image. According to the attribute information of the N objects, M preselected boxes of the target image are determined, wherein the number of objects in the jth preselected box is less than the number of objects in the j+1th preselected box. According to the M preselected boxes, a candidate position set of the target image is obtained, and a target cropping position is determined from the candidate position set according to the attribute information of the N objects. The target image is cropped according to the target cropping position to obtain a cropped image. As can be seen, the target cropping position is determined from the candidate position set according to the attribute information of the N objects, so that the cropped image can include as many key objects (such as objects with a weight higher than a weight threshold) in the N objects as possible and as completely as possible, and the target image can be efficiently processed. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1a A scene schematic diagram of image processing provided by an embodiment of the present application;

[0058] Figure 1b A processing flow schematic diagram of an image processing system provided by an embodiment of the present application;

[0059] Figure 2 A flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0060] Figure 3 A schematic diagram of a preselected box provided by an embodiment of the present application;

[0061] Figure 4 A flow schematic diagram of another image processing method provided by an embodiment of the present application;

[0062] Figure 5a A schematic diagram of extending each edge of a preselected box provided by an embodiment of the present application;

[0063] Figure 5b A schematic diagram of extending an inner edge frame of a box provided by an embodiment of the present application;

[0064] Figure 5c Another schematic diagram of extending an inner edge frame of a box provided by an embodiment of the present application;

[0065] Figure 5d A schematic diagram of a piecewise function provided for an embodiment of the present application;

[0066] Figure 5e A schematic diagram of another piecewise function provided for an embodiment of the present application;

[0067] Figure 6 A structural schematic diagram of an image processing apparatus provided for an embodiment of the present application;

[0068] Figure 7 A structural schematic diagram of an intelligent device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0070] The terms related to the embodiments of the present application will be described below:

[0071] Article: The so-called article refers to an article recommended by a content recommendation platform to a user for reading. The article can contain videos or pictures. The article is usually edited and published on a content platform by a user (such as a self-media) through his / her own account.

[0072] Terminal program: The so-called terminal program refers to an application (such as a social application, a multimedia application, etc.) running on a terminal device, which can accept messages and Feeds stream information.

[0073] Server: The so-called server refers to a server program deployed on one or more groups of servers, which is specially designed to provide remote network services for terminal programs.

[0074] Content recommendation platform: The content recommendation platform can be an application program on a terminal device, a sub-application program (such as a mini-program) in an application program, or a website. The content recommendation platform can recommend multimedia resources such as articles, videos, and music to users for consumption (reading, watching, listening, etc.) based on the latest and hottest contents in the platform and user portraits (user preferences).

[0075] Feeds: message source, also translated as source material, feed, information provider, supply, abstract, source, news subscription, web source (English: web feed, news feed, syndicated feed) is a data format, website through which the latest information is spread to users, usually in a timeline way. Timeline is the most original, direct and basic form of feed display. The prerequisite for users to subscribe to content recommendation platforms is that content recommendation platforms provide message sources. Aggregating feeds is called aggregation, and the software used for aggregation is called aggregator. For end users, aggregators are software specifically designed to subscribe to websites, also known as RSS readers, feed readers, news readers, etc.

[0076] The content recommended by the content recommendation platform (such as articles, short videos, etc.) is usually displayed in the form of Feeds flow for users to quickly refresh. Multimedia platforms, social platforms or news platforms can be regarded as a new type of aggregator, and the subscription source is your friends or public figures / media, and the content is their publicly published dynamics. When the number of friends is large and active, you can receive constantly updated content, which is the most common form of Feed. Time is the ultimate dimension followed by Feed, because the update of content is the result of constantly requesting the server. In practical application, various self-media account creators create their own articles and content, users can subscribe to the content of these articles, and then when the content of the article is updated, the corresponding consulting content is pushed to the user through the message mode of Business to Customer (B2C) Downstream Feeds, of course, users can also actively refresh Feeds to get the latest consulting information.

[0077] Social network: social network originated from network socialization. The starting point of network socialization is e-mail. The Internet is essentially a network of computers. Early E-mail solved the problem of long-distance mail transmission, and to this day it is the most popular application on the Internet. It is also the starting point of network socialization. Network forums (Bulletin Board System, BBS) have taken a step further, making "group sending" and "forwarding" a normal state, theoretically realizing the function of publishing information to all people and discussing topics (the boundary is the number of BBS visitors). It becomes an early Internet content self-generating platform.

[0078] BBS pushed the network socialization one step further, from the cost reduction of pure point-to-point communication to the cost reduction of point-to-face communication. Instant messaging (IM) and blogs are more like upgrades of the previous two social tools, the former improves the instant effect (transmission speed) and the ability to communicate simultaneously (parallel processing); the latter begins to embody the theories of sociology and psychology - the information publishing node begins to embody stronger individual consciousness, because the dispersed information in the time dimension can be aggregated, thus becoming the "image" and "character" of the information publishing node.

[0079] We-media: We-media is different from the information dissemination dominated by professional media organizations. It is an information dissemination activity dominated by the general public, which changes from traditional "point-to-face" dissemination to "point-to-point" dissemination. At the same time, it also refers to an information dissemination method that provides information production, accumulation, sharing, and dissemination content with privacy and publicity. The content of we-media is special, without a fixed core, users can freely express themselves, and share anything they think is valuable. Sometimes, they also share different opinions from the mainstream, so we-media usually attracts users with the same interests with its unique personality.

[0080] Short video: Short video refers to a video whose length is within a certain threshold. It is a way of internet content dissemination, usually referring to video dissemination content on the internet with a length of less than 5 minutes. With the popularity of mobile terminals and the acceleration of the network, short, fast, and high-traffic dissemination content has gradually gained the favor of major platforms, fans, and capital.

[0081] ImageNet: ImageNet is a large visual database for visual object recognition software research. More than 14 million image URLs are manually annotated by ImageNet to indicate the objects in the pictures; in at least one million images, boundary boxes are also provided.

[0082] Artificial Intelligence (AI): AI is the theory, method, technology and application system that use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that the machine has the functions of perception, reasoning and decision making. The embodiments of the present application mainly involve inserting pseudo quantization operators into multiple network layers to be quantized in the model to be trained in stages during the iterative training process to obtain a model to be converted; the model to be converted is converted, and the converted model is trained, and finally a quantization model corresponding to the model to be trained is obtained, so as to reduce the size of the neural network model.

[0083] AI technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, processing technology of large application programs, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0084] Among them, computer vision technology is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, etc. Machine vision, and further image processing, so that computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data; it usually includes image processing, video processing, video semantic understanding, video content / behavior recognition, etc.

[0085] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of AI and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning / deep learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and example-based learning, etc.

[0086] Deep learning: The concept of deep learning is derived from the research of artificial neural networks. Multi-layer perceptron with multiple hidden layers is a deep learning structure. Deep learning forms more abstract high-level representation attribute categories or features by combining low-level features to discover distributed feature representation of data.

[0087] The popular deep learning algorithms can be divided into two categories. One is the R-CNN series algorithm based on Region Proposal (R-CNN, Fast R-CNN, Faster R-CNN), which is two-stage and needs to use a heuristic method (selective search) or a CNN network (RPN) to generate Region Proposal, and then perform classification and regression on the Region Proposal. CNN (Convolutional Neural Networks) is one of the representative algorithms of deep learning. It is a kind of feedforward neural network with convolution calculation and deep structure, which has representation learning ability and can perform shift-invariant classification on input information according to its hierarchical structure. The other category is Yolo, SSD, etc. one-stage algorithm, which only uses a CNN network to directly predict the category and position of different targets.

[0088] The accuracy of the first category of algorithms is higher than that of the second category of algorithms, but the speed is slower. The speed of the second category of algorithms is faster than that of the first category of algorithms, but the accuracy is lower. In the subsequent description of the present application, the object detection model adopts Yolo algorithm as an example for description. The basic idea of Yolo algorithm is to divide the image into multiple grids first, and then perform object detection around each grid. The full name is You Only Look Once: Unified, Real-Time Object Detection. The algorithm characteristics are summarized as follows: You Only Look Once is used to indicate that the algorithm only needs to perform one CNN operation, Unified is used to indicate that this is a unified framework, providing end-to-end prediction, and Real-Time is used to reflect the fast speed of Yolo algorithm.

[0089] Based on computer vision technology, machine learning technology, and deep learning technology in AI technology, the embodiment of the present application provides an image processing scheme to better process an independent image or each frame of image in a video. Figure 1aA scene diagram of image processing is provided for the embodiments of the present application. As shown in Figure 1a The image processing scheme provided by the present application can be executed by the intelligent device 101. The intelligent device 101 can be a terminal or a server with image processing capability. The terminal can include, but is not limited to, a smart phone (such as an Android phone, an IOS phone, etc.), a tablet computer, a portable personal computer, a mobile Internet device (Mobile Internet Devices, MID), etc. The embodiments of the present application do not limit the terminal. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc. The embodiments of the present application do not limit the server.

[0090] In a specific implementation, the flow of the image processing scheme mainly includes: (1) an intelligent device acquires a target image, which can be a single image (such as a photo) or a frame image in a video (for example, the first frame in a video, a key frame, etc.); after acquiring the target image, the intelligent device can call an object detection model (such as a region convolutional neural network (R-CNN), a YOLO (You Only Look Once) algorithm, etc.) to perform feature extraction on the target image, and obtain attribute information of N objects contained in the target image (such as coordinates of an area where an object is located, a confidence (that is, a probability value output by the object detection model for the corresponding object), an object label, a weight, coordinates of key points of the object, etc.), N being a positive integer; (2) the intelligent device determines M pre-selected frames of the target image according to the attribute information of the N objects detected by the object detection model, wherein the number of objects in the jth pre-selected frame is less than the number of objects in the (j+1)th pre-selected frame, j and M are positive integers, j is less than M, and M is less than or equal to N; for example, the intelligent device can merge the N objects in turn according to the distance between the center point of each object and the center point of the target image, and obtain a pre-selected frame each time; for another example, the intelligent device can merge the N objects in turn according to the weight of each object, and obtain M pre-selected frames; (3) the intelligent device obtains a candidate position set of the target image according to the M pre-selected frames, specifically, if the scale of a pre-selected frame matches a target cropping scale, the pre-selected frame is moved in the target image to obtain a plurality of candidate positions corresponding to the pre-selected frame; if the scale of a pre-selected frame does not match the target cropping scale, the pre-selected frame is adjusted based on the target cropping scale, and the adjusted pre-selected frame is moved in the target image to obtain a plurality of candidate positions corresponding to the adjusted pre-selected frame, and the candidate positions are added to the candidate position set. After obtaining the candidate position set, the intelligent device determines a target cropping position from the candidate position set according to the attribute information of the N objects; for example, the score of each candidate position is calculated through the objects included in each candidate position, and then the target cropping position is determined. (4) the intelligent device performs cropping processing on the target image through the target cropping position, and obtains a cropped image.

[0091] Figure 1b A processing flow diagram of an image processing system provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the processing flow of the image processing system mainly includes the following steps. Figure 1bAs shown, the image processing system can include one or more of the following: a content database, a content distribution outlet service, a multi-object detection picture library, a picture subject object labeling service, a manual review system, a dispatch center service, an uplink and downlink content interface server, an intelligent screenshot service, a content deduplication service, a picture subject object detection model, a content analysis and frame extraction service, a download file system, and a content storage service. The image processing method provided in the embodiments of the present application is mainly executed by the intelligent screenshot service in the image processing system. The main functions of each module in the image processing system are introduced as follows: Figure 1b

[0092] On the one hand, the uplink and downlink content interface server receives the content (such as photos, short videos, etc.) uploaded / published by the content production end. Specifically, the content producer of professional generated content (PGC), user generated (original) content (UGC), multi-channel network (MCN), or professional user generated content (PUGC) provides local or shot video content or written WeMedia articles or picture sets through a mobile terminal or a backend application programming interface (API) system. The author can choose to actively upload the cover picture of the corresponding content, and these are the main sources of distributed content.

[0093] The content production end communicates with the uplink and downlink content interface service to first obtain the upload server interface address, and then uploads the local file (i.e., uploads and publishes the content). Before uploading, the content production end can select the matching music, filter template, and beautification function for the uploaded content (such as a video).

[0094] ​The uplink and downlink content interface service and the content production end directly communicate. After receiving the content uploaded by the content production end (usually including one or more of the following: title of the content, identification of the publisher, abstract of the content, cover picture, publishing time, and video to be published), the uplink and downlink content interface service directly enters the server through the server, stores the file into the video content storage service (i.e., stores the source file), and writes the meta information of the video content (such as video file size, cover picture link, code rate, file format, title, publishing time, author, and the like) into the content database (i.e., writes the content meta information). Then, the uplink and downlink content interface service submits the content uploaded by the content production end and the content meta information to the dispatch center service (i.e., the content like dispatch center) for subsequent content processing and circulation. In addition, the duplicate detection service is called to detect the content uploaded by the content production end (i.e., whether there is content in the database that is similar to the content uploaded by the content production end and exceeds the similarity threshold).

[0095] The dispatch center service is responsible for the entire scheduling process of video and text content circulation. The dispatch center service receives the content uploaded by the content production end through the uplink and downlink content interface server, and then obtains the meta information of the content from the content database (i.e., obtains the meta information). As the actual dispatch controller of the text and video link, the dispatch center service dispatches the intelligent screenshot service matching the type of the content (such as animal video, person video, and landscape video) to process the picture content in the link (such as calling the intelligent screenshot service to intercept the key frame in the video). In the scheduling process, the dispatch center service can dispatch the manual review system and the machine processing system to control the order and priority of the dispatch (such as preferentially processing the content uploaded by the authenticated user). In addition, the dispatch center service synchronizes the content uploaded by the content production end to the manual review system. The content processing of the dispatch center mainly includes machine processing and manual review processing. Here, the machine processing core includes various quality judgments (such as low-quality filtering (keeping the video with high definition for the same content)), content labels (such as content classification), label information, and content duplicate detection. The results will be written into the content database (i.e., update the meta information), and the completely repeated content will not be synchronized to the manual review system for repeated secondary processing.

[0096] The manual review system is usually a global web system that takes the results of machine filtering on the link, manually confirms and reviews the results (annotates multiple object data in the business scenario, needs to be annotated by manual review service, and the annotated results are saved in the multi-object detection picture library). During the manual review process, the meta information in the content database is read, and the results and status of the manual review are also fed back into the content database, that is, the review results are written into the content information meta database. In addition, the actual effect of machine strikes (such as deleting illegal content) and filtering models (such as quality screening of the same content) can be evaluated online through the results of manual review.

[0097] The content database is the core database of the content. The meta information of all the content published by the producers is saved in this business database. The focus is on the meta information of the content itself, such as file size, cover picture link, code rate, file format, title, release time, author, video file size, video format, originality mark or first release, and also includes the classification of the content in the manual review process (including one, two, and three level classification and tag information, such as a content explaining XX brand mobile phone, one level subcategory is technology, two level classification is smart phone, three level classification is domestic mobile phone, and tag information is XX brand mobile phone).

[0098] In response to the content in the manual review system being enabled (i.e. passing the review), the content index information obtained by the content consumer, that is, the terminal, is provided by the content export distribution service (usually a recommendation engine or a search engine or operation) directly on the display page. In short, after the content passes the review, it will be recommended to users who may be interested in the content.

[0099] On the other hand, for the content consumer, as a content consumer, the content consumer communicates with the content distribution export server to obtain the index information of the corresponding content (such as news, short video, etc.). After the terminal consumer obtains the content index information, it can also directly access the video content storage server to download the corresponding content. If it is a video content, the content consumer communicates with the video storage server to download the corresponding streaming media file and plays it through the local player. For text and pictures, it usually communicates with the edge deployed CDN service. During the content consumption process, the content consumer reports the user's browsing behavior data, reading speed, completion rate, reading time, stuttering, loading time, play click, etc. to the server. In practical applications, the content consumer usually browses and consumes data through the Feeds flow mode. For low-quality content of the consumer end, a direct reporting and feedback entrance is provided to directly interface with the manual review system for confirmation and review.

[0100] The content storage service is usually a set of widely distributed storage servers. During content consumption, the user on the consumption side accesses the nearby storage servers. The storage servers are usually surrounded by CDN acceleration servers for distributed cache acceleration. The content producer uploads video and picture content through the uplink content interface server, which saves the content. In addition to being a data source for external services, the content storage service is also a data source for internal services, which provides the original video data for the download file system to perform related processing. The internal and external data sources are usually deployed separately to avoid mutual influence.

[0101] For the download file system, the download file system downloads and obtains the original video content (i.e., downloads the video file) from the content storage server and controls the speed and progress of downloading the (video) file from the content storage service. The download file system is usually a set of parallel servers composed of related task scheduling and distribution clusters. The downloaded file calls the frame extraction service to obtain the necessary video file key frame from the video source file, which is used as the basis for subsequent construction of the video fingerprint (a software identification, extraction, and compression technology for videos, which can generate a unique "fingerprint" to represent a video file).

[0102] For the content analysis and frame extraction service, the download file system downloads the file from the video content storage service to preprocess the video file features, that is, to process the source file (such as video frame extraction). The preprocessed pictures can include key frames and uniformly extracted frames. Preprocessing of the video usually refers to extracting the key frames of the video. If the source file is a graphic text content, the graphic text file needs to be parsed, and each picture is parsed from the content to obtain the access address of the picture (i.e., read meta information).

[0103] After the content analysis and frame extraction service processes the source file, the picture main object labeling service obtains the pictures obtained after the content analysis and frame extraction service processes the source file (i.e., reads the labeled source data) and labels the pictures with multiple object data. Through the sample data of the multi-main object detection of the pictures that meet the information flow business, the multi-main object detection of the pictures is constructed. The core target is the face, the person, and the common main body such as animals, vehicles, etc. The specific number of independent main bodies involved in the sample data depends on the number of types (usually more than 60 types according to product labeling and classification types in the information flow graphic text business, which can be continuously expanded and supplemented as needed). The picture data labeling result needs to be audited by the manual auditing service (i.e., push manual auditing). The data labeling result that passes the audit is saved in the multi-object detection picture library (i.e., write main object sample).

[0104] ​​​​Further, the picture subject object detection model (including one or more of the object detection model, the face detection model, and the text recognition model) reads samples from the multi-object detection picture library and optimizes (i.e., model training) the model parameters based on the samples. In an implementation, a part of the sample annotation data of the object detection in the business scenario is supplemented (such as data after data augmentation), and the existing data in the database is combined to manually mark the picture subject, the annotation data is supplemented to the sample data set, and the picture subject object detection model (such as a model based on YOLOv5) is trained through the sample data set. Optionally, the model structure and hyperparameters are not adjusted during the training process.

[0105] After the training is completed, the picture subject object detection model is called to process the picture output by the content analysis and frame extraction service, and attribute information of N objects contained in the picture is output, where N is a positive integer. The intelligent screenshot service responds to the scheduling of the scheduling center service, performs intelligent cropping processing on the picture based on the attribute information of the N objects (for details, refer to the main process of image processing in Figure 1a , which will not be described here again), to obtain a cropped image.

[0106] In the embodiments of the present application, a target image is obtained, and feature extraction is performed on the target image to obtain attribute information of N objects contained in the target image. According to the attribute information of the N objects, M preselected frames of the target image are determined, where the number of objects in the jth preselected frame is less than the number of objects in the j+1th preselected frame. According to the M preselected frames, a candidate position set of the target image is obtained, and a target cropping position is determined from the candidate position set according to the attribute information of the N objects. The target image is cropped according to the target cropping position to obtain a cropped image. As can be seen, the target cropping position is determined from the candidate position set according to the attribute information of the N objects, which can make the cropped image as much as possible and as complete as possible to include key objects (such as objects with a weight higher than a weight threshold) in the N objects, and further enable efficient processing of the target image, so that the cropped image can better attract the attention of users and improve user stickiness.

[0107] Please refer to Figure 2 , Figure 2 for a flowchart of an image processing method provided by the embodiments of the present application. The method of the embodiments of the present application is applied to a smart device, which can be a terminal device used by some users, or a server with special functions. The method includes the following steps:

[0108] S201, a target image is obtained, and feature extraction is performed on the target image to obtain attribute information of N objects contained in the target image.

[0109] The target image can be a single image (such as a photo, an attached figure in an article, etc.) or a frame image in a video (such as the first frame in a video, a key frame, etc.). After obtaining the target image, the intelligent device performs feature extraction on the target image to obtain attribute information of N objects contained in the target image, N being a positive integer. The attribute information of the objects can include, but is not limited to, coordinates of a region where the object is located, a confidence (i.e., a probability value output by the object detection model for the object), an object label (such as a category to which the object belongs), a weight of the object, coordinates of a key point of the object, etc. The attribute information of the objects can be obtained in the manner of the image processing system in Figure 1b , which will not be described herein again.

[0110] In an implementation, the intelligent device can call the object detection model to process the target image to obtain attribute information of N objects output by the object detection model. The object detection model can be a model based on a deep learning algorithm (such as a model based on the YOLO algorithm) or a model composed of multiple sub-models (such as a model composed of a multi-object detection sub-model and a face detection sub-model). Further, the object detection model can be obtained by training an initialized model by a sample data set by the intelligent device.

[0111] Specifically, the intelligent device can obtain labeled data (such as labeled data obtained from ImageNet), and then obtain a sample data set for training the object detection model. The sample data is input into an initialized model to obtain prediction data output by the initialized model. A loss value corresponding to a loss function of the initialized model is calculated based on the labeled data and the prediction data. The parameters in the initialized model are adjusted by a gradient descent method until the loss value is less than a loss threshold, and the object detection model is obtained.

[0112] S202, determining M pre-selected frames of the target image according to the attribute information of the N objects.

[0113] Each pre-selected frame includes at least one object of the N objects. The pre-selected frame can be determined based on position information of at least one object randomly selected from the N objects, based on a distance between a center point of each object and a center point of the target image (such as position information of at least one object with a distance less than a distance threshold), or based on a weight of each object (such as position information of at least one object with a weight greater than a weight threshold). M is a positive integer, and M is less than or equal to N.

[0114] In one implementation, the number of objects in the j-th preselection box is less than the number of objects in the (j+1)-th preselection box, where j is a positive integer and j is less than M; for example, the j-th preselection box includes P objects, where P is a positive integer and P is less than N, and the (j+1)-th preselection box is constructed based on the fact that the j-th preselection box includes P objects, and at least one of the N objects other than the P objects included in the j-th preselection box, that is, the j-th preselection box is contained in the (j+1)-th preselection box.

[0115] Figure 3 This is a schematic diagram of a preselection box provided in an embodiment of this application. Figure 3 As shown, let the vertex coordinates of object 1 be (1,1), (1,3), (4,1), (4,3) corresponding to region 1, and the vertex coordinates of object 2 be (3,5), (7,5), (3,9), (7,9) corresponding to region 2. The first preselection box only includes object 1, and the second preselection box includes both object 1 and object 2. Then the first preselection box includes region 1 but does not include region 2, and the second preselection box includes both region 1 and region 2.

[0116] S203. Based on the M preselected boxes, obtain the candidate position set of the target image, and determine the target cropping position from the candidate position set based on the attribute information of N objects.

[0117] Each preselected bounding box corresponds to one or more candidate locations (when the scale of the preselected bounding box is the same as the scale of the target image, it corresponds to one candidate location (i.e., the target image itself); when the scale of the preselected bounding box is smaller than the scale of the target image, it corresponds to multiple candidate locations). The intelligent device can determine a candidate location based on the current position of each preselected bounding box, and then slide each preselected bounding box across the target image to determine multiple candidate locations corresponding to each candidate bounding box, thus obtaining a set of candidate locations for the target image.

[0118] In an embodiment, the intelligent device obtains a target cropping size, which can be preset or determined according to actual requirements (e.g., the size of the picture required for displaying the picture on device 1 is A*B, and the target cropping size is set as A*B; the size of the picture required for displaying the picture on device 2 is C*D, and the target cropping size is set as C*D). For any preselected frame, if the size of the preselected frame matches the target cropping size, the preselected frame is moved in the target picture to obtain a plurality of candidate positions corresponding to the preselected frame; if the size of the preselected frame does not match the target cropping size, the preselected frame is adjusted (the width or height of the preselected frame is reduced so that the aspect ratio of the adjusted preselected frame matches the aspect ratio of the target cropping size) based on the target cropping size, and the adjusted preselected frame is moved in the target picture to obtain a plurality of candidate positions corresponding to the adjusted preselected frame, and the candidate positions are added to the candidate position set.

[0119] After obtaining the candidate position set, the intelligent device determines the target cropping position from the candidate position set according to the attribute information of the N objects. The intelligent device can score each candidate position according to one or more of the following factors: (1) the score of each candidate position is calculated according to the positional relationship between each candidate position and the frame of the target picture; (2) the score of each candidate position is calculated according to the weight of the object included in each candidate position; (3) the score of each candidate position is calculated according to the integrity of the object included in each candidate position, etc. After calculating the scores of each candidate position in the candidate position set, the intelligent device can determine the candidate position with the highest score or the candidate position with a score higher than a score threshold as the target cropping position.

[0120] S204, cropping the target picture according to the target cropping position to obtain a cropped picture.

[0121] In an embodiment, after determining the target cropping position, the intelligent device can directly move the cropping frame to the target position in the target picture according to the coordinates of the target position, and cut out the image content included in the cropping frame at the target position from the target picture to obtain a cropped picture. Further, the intelligent device can scale the cropped picture according to the target cropping size to output a scaled picture (e.g., the intelligent device directly displays the scaled picture, or sends the scaled picture to a display device for display).

[0122] In the embodiment of the present application, a target image is acquired, and feature extraction is performed on the target image to obtain attribute information of N objects contained in the target image. According to the attribute information of the N objects, M preselected boxes of the target image are determined, wherein the number of objects in the jth preselected box is less than the number of objects in the j+1th preselected box. According to the M preselected boxes, a candidate position set of the target image is obtained, and a target cropping position is determined from the candidate position set according to the attribute information of the N objects. The target image is cropped according to the target cropping position to obtain a cropped image. As can be seen, the target cropping position is determined from the candidate position set according to the attribute information of the N objects, so that the cropped image can include as many key objects (such as objects with a weight higher than a weight threshold) in the N objects as possible and as completely as possible, and the target image can be efficiently processed, so that the cropped image can better attract the attention of users and improve user stickiness.

[0123] Referring to Figure 4 , Figure 4 The flowchart of another image processing method provided by the embodiment of the present application is shown. The method of the embodiment of the present application is applied to a smart device, which may, for example, be a terminal device used by some users or a server with special functions. The method comprises the following steps:

[0124] S401, a target image is acquired, and feature extraction is performed on the target image to obtain attribute information of N objects contained in the target image.

[0125] The specific implementation of step S401 can refer to the implementation of step S201 in Figure 2 , which will not be described here again.

[0126] The attribute information of the N objects can be recorded in the form of a list, a set, a key-value pair or a matrix. In an embodiment, the smart device maintains an attribute information list and an object information list based on the attribute information of the N objects (i.e., records the attribute information of the N objects in the attribute information list and the object information list). The object attribute list records the identification (such as the serial number) of the N objects, the position coordinates, the confidence (i.e., the probability value of the object detection model outputting the corresponding object), the object label (such as the category to which the object belongs), the weight of the object, the coordinates of the key points of the object, etc.; and the object information list records the identification and name of the N objects. Table 1 is an object attribute list provided by the embodiment of the present application.

[0127] Table 1

[0128]

[0129]

[0130] As shown in Table 1, the object attribute list records attribute information of N objects, and the attribute information includes object identification, position coordinates, confidence, object label, weight, key point coordinates, and the like (which can be extended according to actual needs, such as display priority). The object identification has an indexing function. For example, through the object identification "1", it can be determined that the position coordinates of the object are "(1, 1), (1, 3), (4, 1), (4, 3)", the confidence is "0.9", the object label is "pet", the weight is "50", and the key point coordinates are "(2, 2)". The data in Table 1 (such as "animal", "(2, 2)", and the like) is only used for example and does not constitute the actual limitation of the application.

[0131] Table 2 is an object information list provided by an embodiment of the application.

[0132] Table 2

[0133] Object identification Object name … 1 Dog … … … … N … …

[0134] As shown in Table 2, the object information list records object information of N objects, and the object information includes object name (which can be extended according to actual needs, such as whether the orientation is front) and the like. The object identification has an indexing function. For example, through the object identification "1", it can be determined that the object name of the object is "dog". In addition, the object attribute list and the object information list can be associated through the object identification. For example, according to the association of Table 1 and Table 2, the position coordinates of the "dog" are "(1, 1), (1, 3), (4, 1), (4, 3)", the confidence is "0.9", the object label is "pet", the weight is "50", and the key point coordinates are "(2, 2)".

[0135] In actual application, the object information of the N objects is respectively stored in the object attribute list and the object information list, which can facilitate maintenance and extension. Alternatively, the intelligent device can maintain the object information of the N objects in one table.

[0136] S402, determining the line distance between the key point of each object and the center point of the target image according to the attribute information of the N objects, and sorting the N objects according to the line distance of each object to obtain an object sorting set.

[0137] The key points can include, but are not limited to, center points, for example, for a person, the key points are located in a face region. In an implementation, the attribute information of the N objects includes position information (such as coordinates) of the key points corresponding to the N objects respectively. The intelligent device determines N clue boxes corresponding to the N objects based on the object attribute list (determined by the position coordinates of each object), generates a clue box list, and in the clue box list, the intelligent device determines the key points in each clue box according to the types of different objects (for example, the key points in the clue box where an animal is refer to the center points, the key points in the clue box where a face is refer to the points on the outline of the five features, and the key points in the clue box where a text is refer to the center points). The intelligent device calculates the distances between the key points of the N objects and the center point of the target image according to the position information of the key points of the N objects, sorts the N objects in ascending order of the distances, and obtains an object sorting set. In the object sorting set, the distance between the key point of the i th object and the center point of the target image is less than or equal to the distance between the key point of the i+1 th object and the center point of the target image, i is a positive integer, and i is less than N. The object sorting set can be the object information list in step S401 or the object attribute list in step S401.

[0138] Further, the intelligent device can set (update) the weights of the objects or set configuration parameters of the weights of the objects based on the distances between the key points of the objects and the center point of the target image, and obtain an updated clue box list; for example, assuming that the target image includes a puppy 1 and a puppy 2, and the initial weights are both 1, the puppy 1 is at the center of the target image, and the puppy 2 is at the edge of the target image, the intelligent device can update the weight of the puppy 1 from 1 to 2 and update the weight of the puppy 2 from 1 to 0.2, or set the configuration parameter of the puppy 1 to 2 and set the configuration parameter of the puppy 2 to 0.2.

[0139] Optionally, the weights of the objects are updated according to the object information; for example, assuming that the target image includes a star and an ordinary person, and the initial weights are both 1, the intelligent device can set the weight of the star to 5 and set the weight of the ordinary person to 0.5.

[0140] S403, determining M preselected boxes of the target image according to the object sorting set.

[0141] The intelligent device determines the j th preselected box of the target image according to the position information of the first P objects in the object sorting set. The j th preselected box includes the first P objects in the object sorting set, and the ratio of the area of the j th preselected box to the area of the target image is greater than a first proportion threshold, P and j are positive integers, and P is less than or equal to N and j is less than or equal to M.

[0142] In an embodiment, the intelligent device merges the N objects in sequence according to the order in the object ordering set, and obtains an object merging frame after each merging; for example, merging object 1 and object 2 to obtain object merging frame 1, and merging object 3 and object merging frame 1 to obtain object merging frame 2. After each object merging frame is obtained, the intelligent device checks whether the ratio of the area of the merging frame to the area of the target image is greater than a first ratio threshold (such as 40%), until the ratio of the area of the current object merging frame to the area of the target image is greater than the first ratio threshold, and the intelligent device determines the current object merging frame as the first preselected frame. If the first preselected frame is obtained by merging the first P-3 objects in the object ordering set, the second preselected frame is obtained by merging the first P-2 objects in the object ordering set, and the M-1th preselected frame is obtained by merging the N objects in the object ordering set. The Mth preselected frame is obtained based on the target image (i.e., taking the scale of the target image as the scale of the Mth preselected frame). If the ratio of the area of the merging frame obtained by merging the N objects to the area of the target image is still less than the first ratio threshold, the merging frame obtained by merging the N objects is determined as the preselected frame.

[0143] S404, obtaining a target cropping scale, and adjusting the scales of the M preselected frames based on the target cropping scale to obtain M candidate frames of the target image.

[0144] The target cropping scale can be preset or determined according to actual needs (for example, the scale of the picture required when displaying the picture on device 1 is A*B, and the target cropping scale is set to A*B; the scale of the picture required when displaying the picture on device 2 is C*D, and the target cropping scale is set to C*D). For any preselected frame, if the scale of the preselected frame matches the target cropping scale, the preselected frame is moved in the target image to obtain multiple candidate positions corresponding to the preselected frame; if the scale of the preselected frame does not match the target cropping scale, the preselected frame is adjusted based on the target cropping scale (reducing the width or height of the preselected frame so that the aspect ratio of the adjusted preselected frame matches the aspect ratio of the target cropping scale), and the adjusted preselected frame is moved in the target image to obtain multiple candidate positions corresponding to the adjusted preselected frame, and the candidate positions are added to the candidate position set.

[0145] In the specific implementation process, the specific implementation of the intelligent device adjusting the scales of the M preselected frames based on the target cropping scale to obtain M candidate frames of the target image can include the following steps S4041-S4046:

[0146] S4041: determining an extension frame corresponding to the kth preselected frame according to the position relationship between the kth preselected frame and the N objects, k being a positive integer and k being less than or equal to M.

[0147] In an embodiment, the image intercepted by the kth preselected frame includes Q objects, Q is a positive integer, and Q is less than N; the intelligent device expands each edge of the kth preselected frame outward to obtain an expansion frame corresponding to the kth preselected frame. The ratio of the area of the image intercepted by the expansion frame corresponding to the kth preselected frame that overlaps the (Q+1)th object to the area of the (Q+1)th object is less than or equal to a second proportion threshold (for example, 10%).

[0148] Figure 5a An embodiment of the present application provides a schematic diagram of expanding each edge of a preselected frame. As shown in Figure 5a the target image includes objects 1-6, and the distance of objects 1-6 from the center point of the target image increases in turn, the preselected frame 1 includes objects 1 and 2, and each edge of the preselected frame 1 is expanded outward to obtain an expansion frame 1 corresponding to the preselected frame 1, and the ratio of the area of the expansion frame 1 that overlaps object 3 to the area of object 3 is less than or equal to a second proportion threshold; similarly, the ratio of the area of the expansion frame 1 that overlaps object 4 to the area of object 4 is less than or equal to a second proportion threshold; the ratio of the area of the expansion frame 1 that overlaps object 5 to the area of object 5 is less than or equal to a second proportion threshold; and the ratio of the area of the expansion frame 1 that overlaps object 6 to the area of object 6 is less than or equal to a second proportion threshold. Further, after obtaining the expansion frame 1 corresponding to the preselected frame 1, the intelligent device generates a preselected frame 2 on the basis of the preselected frame 1 according to the distance of objects 1-6 from the center point of the target image. For details, refer to the embodiment of determining the M preselected frames in step S403, which will not be described here. Similarly, each edge of the preselected frame 2 is expanded outward to obtain an expansion frame 2 corresponding to the preselected frame 2. Since there is no object above the preselected frame 2, the upper edge of the expansion frame 2 coincides with the edge of the target image, and the ratio of the area of the expansion frame 2 that overlaps object 4 to the area of object 4 is less than or equal to a second proportion threshold; the ratio of the area of the expansion frame 2 that overlaps object 5 to the area of object 5 is less than or equal to a second proportion threshold; and the ratio of the area of the expansion frame 2 that overlaps object 6 to the area of object 6 is less than or equal to a second proportion threshold.

[0149] S4042: determining a first inner edge frame and a second inner edge frame corresponding to the kth preselected frame according to the distance of the expansion frame corresponding to the kth preselected frame from the edge of the target image, wherein the area of the first inner edge frame is less than the area of the second inner edge frame.

[0150] The intelligent device determines the inner edge frame of the expansion frame corresponding to the kth preselected frame according to the positional relationship between each edge of the expansion frame corresponding to the kth preselected frame and each edge of the target image. Figure 5b An embodiment of the present application provides a schematic diagram of the inner edge frame of an expansion frame. As shown in Figure 5bAs shown, the inner edges of the extension frame include inner edge 1-inner edge 4. In an embodiment, the first inner edge refers to the inner edge with the smallest scale (area) among the inner edges of the extension frame corresponding to the kth preselected frame, and the second inner edge refers to the inner edge with the largest scale (area) among the inner edges of the extension frame corresponding to the kth preselected frame.

[0151] In an embodiment, the intelligent device compares the area (scale) of the target cropping scale with the areas between the first inner edge and the second inner edge; if the area of the target cropping scale is greater than the area of the first inner edge and less than the area of the second inner edge, step S4043 is performed; if the area of the target cropping scale is less than or equal to the area of the first inner edge, step S4044 is performed; and if the area of the target cropping scale is greater than or equal to the area of the second inner edge, step S4045 is performed.

[0152] In another embodiment, the target image belongs to a target video, and the intelligent device first determines whether the target image is a front y-frame image of the target video (e.g., whether the target image is a first frame image), where y is a positive integer; if the target image is not a front y-frame image of the target video, the processing of the target image can refer to the above embodiment, which will not be repeated here.

[0153] If the target image is a front y-frame image of the target video, edge detection is performed on the target image (to detect whether there is a text object (e.g., a title of a short video) in the edge of the target image), taking the upper edge of the target image as an example, the edge of the upper part of the target image refers to the area where the first z rows of pixel points from the top of the target image are located, where z is a positive integer; in response to the existence of a text object in the edge area of the target image, a third inner edge is determined based on the position of the text object, and the third inner edge includes the text object; and based on the distance between the extension frame corresponding to the kth preselected frame and the edge of the target image and the third inner edge, the first inner edge (the minimum inner edge corresponding to the kth preselected frame) and the second inner edge (the maximum inner edge corresponding to the kth preselected frame) corresponding to the kth preselected frame are determined. Figure 5c Another schematic diagram of the inner edges of the extension frame provided by an embodiment of the present application is shown in FIG. 4B. Figure 5c As shown, the inner edges of the extension frame include inner edge 1-inner edge 4. If the area of the target cropping scale is greater than the area of the first inner edge and less than the area of the second inner edge, step S4046 is performed; the embodiments in which the area of the target cropping scale is less than or equal to the area of the first inner edge and the area of the target cropping scale is greater than or equal to the area of the second inner edge can refer to the above embodiment, which will not be repeated here.

[0154] S4043: Adjust the aspect ratio of the kth preselected frame according to the aspect ratio of the target cropping scale to obtain the kth candidate frame.

[0155] In an embodiment, the width of the kth preselected frame is kept unchanged, and the height of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches (e.g., is the same as) the aspect ratio of the target cropping scale. In another embodiment, the height of the kth preselected frame is kept unchanged, and the width of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the target cropping scale.

[0156] Optionally, the width and the height of the kth preselected frame are simultaneously adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the target cropping scale.

[0157] S4044: Adjust the aspect ratio of the kth preselected frame according to the aspect ratio of the first inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame.

[0158] In an embodiment, the width of the kth preselected frame is kept unchanged, and the height of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the first inner border. In another embodiment, the height of the kth preselected frame is kept unchanged, and the width of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the first inner border.

[0159] The specific implementation of the adaptive processing of the kth preselected frame by the intelligent device according to the target cropping scale can refer to step S4043, and will not be repeated here.

[0160] S4045: Adjust the aspect ratio of the kth preselected frame according to the aspect ratio of the second inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame.

[0161] In an embodiment, the width of the kth preselected frame is kept unchanged, and the height of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the second inner border. In another embodiment, the height of the kth preselected frame is kept unchanged, and the width of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the second inner border.

[0162] The specific implementation of the adaptive processing of the kth preselected frame by the intelligent device according to the target cropping scale can refer to step S4043, and will not be repeated here.

[0163] S4046: Adjust the aspect ratio of the kth preselected frame according to the aspect ratio of the third inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame.

[0164] In an implementation, the width of the kth preselected frame is kept unchanged, and the height of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the third inner border. In another implementation, the height of the kth preselected frame is kept unchanged, and the width of the kth preselected frame is adjusted so that the adjusted aspect ratio of the kth preselected frame matches the aspect ratio of the third inner border.

[0165] The specific implementation of the smart device in adapting the kth preselected frame according to the target cropping scale can refer to step S4043, which will not be repeated here.

[0166] S405, sliding each candidate frame in the target image to obtain a candidate position set of the target image.

[0167] The smart device slides each candidate frame in a preset direction (e.g., from left to right, from top to bottom) to obtain at least one candidate position corresponding to each candidate frame, and further obtain a candidate position set of the target image. Each candidate position is determined according to the position of the candidate frame in the target image after each sliding. The sliding distance of the position of the candidate frame after each sliding compared to the previous position can be a preset distance (e.g., in pixel points, the sliding distance of the position of the candidate frame after sliding compared to the previous position is R pixel points, and R is a positive integer). It can be understood that the smaller the distance of each sliding, the more candidate positions of the candidate frame included in the target image of the same scale, and therefore the sliding distance can also be determined according to influencing factors (e.g., the size of the target image, the performance of the server, etc.).

[0168] S406, calculating the cumulative score of each candidate position in the candidate position set according to the attribute information of the N objects.

[0169] In an implementation, the cumulative score of each candidate position is obtained by summing the first score and the second score of the candidate position, the first score is calculated based on the position of the candidate position in the target image, and the second score is calculated based on the object included in the candidate position. The xth candidate position has an overlapping region with at least one object, and the target object is any one of the at least one object.

[0170] On the one hand, the smart device calculates the first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, x is a positive integer, and x is less than or equal to M. Specifically, the smart device calculates the first area score of the xth candidate position according to the area relationship between the target cropping scale and the first inner border and the second inner border corresponding to the preselected frame corresponding to the xth candidate position, and the calculation formula of the first area score is as follows:

[0171] prop_score = prop_spec / crop_prop

[0172] wherein prop_score is the first area score, and prop_spec is determined according to the area relationship between the first inner border and the second inner border corresponding to the preselected frame corresponding to the target cropping scale and the xth candidate position. If the target cropping scale is less than or equal to the first inner border corresponding to the preselected frame corresponding to the xth candidate position, prop_spec = min_prop; if the target cropping scale is greater than or equal to the second inner border corresponding to the preselected frame corresponding to the xth candidate position, prop_spec = max_prop; in other cases (such as the target cropping scale is greater than the first inner border corresponding to the preselected frame corresponding to the xth candidate position, and less than the second inner border corresponding to the preselected frame corresponding to the xth candidate position), prop_spec = crop_prop. The values of min_prop, max_prop and crop_prop can be set or calculated according to actual requirements; for example, min_prop is the ratio of the area of the first inner border to the area of the target image, max_prop is the ratio of the area of the second inner border to the area of the target image, and crop_prop = 1.

[0173] Then, the smart device calculates the ratio of the area of the xth candidate position to the area of the target image to obtain the second area score of the xth candidate position, and the calculation formula of the second area score is as follows:

[0174] area_prop_score = box_area / org_area

[0175] wherein area_prop_score is the second area score, box_area is the area of the xth candidate position, and org_area is the area of the target image.

[0176] After obtaining the first area score and the second area score, the smart device performs operation processing on the first area score, the second area score and the area of the target image to obtain the first score of the xth candidate position, and the calculation formula of the first score is as follows:

[0177] padding_score = (prop_score + area_prop_score) * org_area / w

[0178] wherein padding_score is the first score (i.e. the inner border score corresponding to the xth candidate position), prop_score is the first area score, area_prop_score is the second area score, org_area is the area of the target image, and w is a configuration parameter, which can be an empirical value, such as w = 4.

[0179] In another aspect, the smart device calculates an overlap area ratio according to the overlap area of the target object and the xth candidate position and the area of the target object, and calculates the object score of the target object according to the overlap area ratio and the weight of the target object. Specifically, the overlap area ratio = overlap area / area of the target object; for example, the overlap area is 0.3 and the area of the target object is 1, then the overlap area ratio = 0.3. After obtaining the overlap area ratio of the target object, the smart device can calculate the overlap proportion score based on a piecewise function, and the calculation formula of the overlap proportion score is as follows:

[0180] overlap_prop_score = f (obj_overlap_prop)

[0181] Wherein, overlap_prop_score is the overlap proportion score, obj_overlap_prop is the overlap area ratio, and f(x) is a piecewise function. Figure 5d A schematic diagram of a piecewise function provided for an embodiment of the present application. The piecewise function can be a piecewise function corresponding to objects such as people, faces, and animals. As shown in Figure 5d When the overlap area ratio is (0, a], the overlap_prop_score = 0; when the overlap area ratio = b, the overlap_prop_score = -d; when the overlap area ratio is [c, 1], the overlap_prop_score = d; wherein 0 < a < b < c < 1, the values of a, b, c and d can be set according to actual conditions (for example, a = 0.1, b = 0.5, c = 0.9, d = 0.5). Figure 5e A schematic diagram of another piecewise function provided for an embodiment of the present application. The piecewise function can be a piecewise function corresponding to text objects and the like. As shown in Figure 5e When the overlap area ratio = e, the overlap_prop_score = -g; when the overlap area ratio = f, the overlap_prop_score = 0; when the overlap area ratio = 1, the overlap_prop_score = g; wherein 0 < e < f < 1, the values of e, f and g can be set according to actual conditions (for example, e = 0.45, f = 0.9, g = 0.45). In addition, if the target object belongs to other types (such as the target object is a car), then overlap_prop_score = obj_overlap_prop.

[0182] Next, the smart device calculates the object score of the target object according to the overlap proportion score and the weight of the target object, and the calculation formula of the object score is as follows:

[0183] score1 = overlap_prop_score * area * conf * weight

[0184] wherein score1 is the object score, overlap_prop_score is the overlap proportion score, area is the area of the target object, conf is the weight configuration parameter of the target object, and weight is the weight of the target object.

[0185] The intelligent device calculates the key point score of the target object according to the coordinates of the key point of the target object. Specifically, the intelligent device evaluates the key point of the target object as follows:

[0186] x_prop = (middle_pos_x - x0) / (x1 - x0), if x_prop > 0.5, x_prop = 1 - x_prop

[0187] y_prop = (middle_pos_y - y0) / (y1 - y0), if y_prop > 0.5, y_prop = 1 - y_prop

[0188] wherein x0 and y0 are the horizontal coordinate and the vertical coordinate of the key point of the target object (if the target object contains multiple key points, the center points of the key points are determined as the key point of the target object); x1 and y1 are the horizontal coordinate and the vertical coordinate of the edge coordinate point of the xth candidate position; middle_pos_x and middle_pos_y are the horizontal coordinate and the vertical coordinate of the center point of the xth candidate position; and x_prop and y_prop represent the evaluation value of the key point of the target object.

[0189] If the key point of the target object is outside the xth candidate position, the calculation formula of the key point score of the target object at the xth candidate position is as follows:

[0190] score2 = -area * conf * (abs(x_prop) + abs(y_prop)) / 2 * weight

[0191] wherein score2 represents the key point score of the target object, area is the area of the target object, conf is the weight configuration parameter of the target object, weight is the weight of the target object, and abs(x) is an absolute value function.

[0192] If the key point of the target object is not outside the xth candidate position, the calculation formula of the key point score of the target object at the xth candidate position is as follows:

[0193] score2 = area * conf * (abs(x_prop) + abs(y_prop)) / 2 * weight

[0194] Wherein, score 2 represents the key point score of the target object, area is the area of the target object, conf is the weight configuration parameter of the target object, weight is the weight of the target object, and abs(x) is an absolute value function.

[0195] The intelligent device calculates the object score and the key point score of the at least one object having the overlapping area with the xth candidate position according to the above method, and then accumulates the object score and the key point score of the at least one object to obtain the second score of the xth candidate position. For example, assuming that the candidate position 1 includes an object 1 and an object 2, the object score of the object 1 is 15, the key point score of the object 1 is 5, the object score of the object 2 is 48, and the key point score of the object 2 is 10, then the second score of the candidate position 1 is 78.

[0196] In an embodiment, the score calculation formula of the candidate box can be represented as:

[0197] core=padding_score+Max(obj_score)

[0198] Wherein, core is the score of the candidate box, padding_score is the first score (the first score of the candidate position corresponding to the same candidate box is the same), and Max(obj_score) is the second score of the candidate position with the highest second score in the candidate position corresponding to the candidate box.

[0199] S407. Determine the candidate position with the accumulated score higher than the score threshold value in the candidate position set as the target cropping position.

[0200] The score threshold value can be an empirical value or set according to actual conditions; for example, the intelligent device can set the score threshold value to determine the candidate position with the highest score in the candidate position set as the target cropping position; for another example, the intelligent device can set the score threshold value to determine the candidate position arranged in the front S (such as 9) in the candidate position set (sorted in descending order of accumulated scores) as the target cropping position, and S is a positive integer.

[0201] S408. Cropping the target image according to the target cropping position to obtain a cropped image.

[0202] The specific implementation of step S408 can refer to the implementation of step S204 in Figure 2 The specific implementation of step S204 in

[0203] In this application, feature extraction is performed on the target image to obtain attribute information of N objects contained in the target image. Candidate boxes are constructed based on the attribute information of these N objects. Then, weights are set for different objects according to their categories and key point positions, thereby obtaining a list of cue boxes (including coordinates, confidence scores, categories, weights, key point lists, etc.). Next, for all pre-selected boxes, the cue list information is used to adapt to the target size, resulting in a set of candidate positions for the target image. The candidate position with the highest score is used as the target cropping position to crop the target image, resulting in the cropped image. Through this application, a deeper understanding of the main content of the image can be achieved. During the cropping process, the attribute information of objects in the image is fully utilized to ensure the integrity of the final screenshot. At the same time, it can improve the generation efficiency of qualified and high-quality adapted cover images, and has a good adaptation effect for display scenarios of various specifications and sizes of content. It can enhance the attractiveness of the final generated cover image to users, improve the conversion efficiency of content clicks, and thus increase consumption stickiness and consumption time.

[0204] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0205] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The apparatus can be mounted on... Figure 1a On the smart device 101 shown. Figure 6 The image processing apparatus shown can be used to perform the above. Figure 2 and Figure 4 The described method embodiments include some or all of the functionalities. The detailed descriptions of each unit are as follows:

[0206] Acquisition unit 601 is used to acquire the target image;

[0207] The processing unit 602 is used to extract features from the target image to obtain attribute information of N objects contained in the target image, where N is a positive integer;

[0208] And to determine M preselection boxes of the target image based on the attribute information of N objects, wherein the number of objects in the j-th preselection box is less than the number of objects in the (j+1)-th preselection box, j and M are positive integers, and j is less than M, and M is less than or equal to N;

[0209] And it is used to obtain a set of candidate positions for the target image based on M preselected boxes, and to determine the target cropping position from the set of candidate positions based on the attribute information of N objects;

[0210] And the target image is cropped by the target cropping position to obtain a cropped image.

[0211] In an embodiment, the attribute information of the N objects includes position information of key points corresponding to the N objects respectively; the processing unit 602 is configured to determine M preselected frames of the target image according to the attribute information of the N objects, and specifically configured to:

[0212] According to the position information of the key points of the N objects, the distances of the key points of the N objects to the center point of the target image are determined, and the N objects are sorted according to the distances to obtain an object sorting set;

[0213] The distance of the key point of the i th object to the center point of the target image is less than or equal to the distance of the key point of the i+1 th object to the center point of the target image, i is a positive integer, and i is less than N;

[0214] According to the position information of the first P objects in the object sorting set, the j th preselected frame of the target image is determined;

[0215] The j th preselected frame includes the first P objects in the image, and the ratio of the area of the j th preselected frame to the area of the target image is greater than a first proportion threshold, P and j are positive integers, and P is less than or equal to N, and j is less than or equal to M.

[0216] In an embodiment, the processing unit 602 is configured to obtain a candidate position set of the target image according to the M preselected frames, and specifically configured to:

[0217] Obtain a target cropping scale, and adjust the scale of the M preselected frames based on the target cropping scale to obtain M candidate frames of the target image;

[0218] Slide each candidate frame in the target image to obtain a candidate position set of the target image;

[0219] The candidate position corresponding to the target candidate frame is the position of the target candidate frame in the target image after each sliding.

[0220] In an embodiment, the processing unit 602 is configured to adjust the scale of the M preselected frames based on the target cropping scale to obtain M candidate frames of the target image, and specifically configured to:

[0221] According to the position relationship between the k th preselected frame and the N objects, an extended frame corresponding to the k th preselected frame is determined, k is a positive integer, and k is less than or equal to M;

[0222] determine a first inner border and a second inner border corresponding to the kth preselected frame according to a distance between the extension frame corresponding to the kth preselected frame and the edge of the target image, the area of the first inner border being smaller than the area of the second inner border;

[0223] if the area of the target cropping scale is greater than the area of the first inner border and smaller than the area of the second inner border, adjust the width-height ratio of the kth preselected frame according to the width-height ratio of the target cropping scale to obtain the kth candidate frame.

[0224] In an implementation, the image intercepted by the kth preselected frame includes Q objects, Q being a positive integer and Q being smaller than N; the processing unit 602 is configured to determine a first inner border and a second inner border corresponding to the kth preselected frame according to a distance between the kth preselected frame and the edge of the target image, and specifically configured to:

[0225] extend each side of the kth preselected frame to obtain an extension frame corresponding to the kth preselected frame;

[0226] The ratio of the area of the Q+1th object to the area of the Q+1th object in the image intercepted by the extension frame corresponding to the kth preselected frame is less than or equal to a second proportion threshold.

[0227] In an implementation, the processing unit 602 is further configured to:

[0228] if the area of the target cropping scale is less than or equal to the area of the first inner border, adjust the width-height ratio of the kth preselected frame according to the width-height ratio of the first inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame;

[0229] if the area of the target cropping scale is greater than or equal to the area of the second inner border, adjust the width-height ratio of the kth preselected frame according to the width-height ratio of the second inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame;

[0230] The scale of the kth candidate frame matches the target cropping scale.

[0231] In an implementation, the attribute information of the N objects includes the weight corresponding to each object and the coordinates of the key points of each object; the processing unit 602 is configured to determine the target cropping position from the candidate position set according to the attribute information of the N objects, and specifically configured to:

[0232] calculate a first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, x being a positive integer and x being less than or equal to M;

[0233] calculate a second score of the xth candidate position according to the weight of each object in the xth candidate position and the coordinates of the key points of each object.

[0234] summing the first score and the second score to obtain a cumulative score of the xth candidate position;

[0235] determining, as the target cropping position, a candidate position in the candidate position set whose cumulative score is higher than a score threshold.

[0236] In an implementation, the processing unit 602 is configured to calculate the first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, and specifically configured to:

[0237] calculate the first area score of the xth candidate position according to the area relationship between the target cropping scale and the first inner frame and the second inner frame corresponding to the preselected frame corresponding to the xth candidate position;

[0238] calculate the second area score of the xth candidate position by calculating the ratio of the area of the xth candidate position to the area of the target image;

[0239] perform operation processing on the first area score, the second area score, and the area of the target image to obtain the first score of the xth candidate position.

[0240] In an implementation, the xth candidate position and at least one object have an overlapping area, and the target object is any one of the at least one object; the processing unit 602 is configured to calculate the second score of the xth candidate position according to the weight of each object in the xth candidate position and the coordinates of the key points of each object, and specifically configured to:

[0241] calculate the object score of the target object according to the overlapping area ratio and the weight of the target object;

[0242] calculate the key point score of the target object according to the coordinates of the key points of the target object;

[0243] perform accumulation operation on the object score and the key point score of the at least one object to obtain the second score of the xth candidate position.

[0244] In an implementation, the target image belongs to a target video; the processing unit 602 is further configured to:

[0245] if the target image is a front y-frame image of the target video, perform edge detection on the target image, y being a positive integer;

[0246] in response to the existence of a text object in the edge region of the target image, determine the third inner frame corresponding to the kth preselected frame, the third inner frame including the text object;

[0247] If the area of the target cropping size is greater than the area of the first inner border and less than the area of the second inner border, the aspect ratio of the kth preselected frame is adjusted according to the aspect ratio of the third inner border, and the kth preselected frame is adapted according to the target cropping size, to obtain the kth candidate frame.

[0248] The kth candidate frame has a size matched with the target cropping size.

[0249] According to an embodiment of the present application, Figure 2 and Figure 4 Part of the steps involved in the image processing method shown in Figure 6 may be performed by each unit in the image processing device shown in Figure 2 For example, step S201 shown in Figure 6 may be performed by the acquisition unit 601 and the processing unit 602 in cooperation, and steps S202-S204 may be performed by the processing unit 602. Figure 6 Step S401 and step S404 shown in Figure 4 may be performed by the acquisition unit 601 and the processing unit 602 in cooperation, and steps S402, S403, S405-S408 may be performed by the processing unit 602. Figure 6 Step S401 and step S404 shown in Figure 6 may be performed by the acquisition unit 601 and the processing unit 602 in cooperation, and steps S402, S403, S405-S408 may be performed by the processing unit 602. Figure 6 Each unit in the image processing device shown in may be combined into one or several other units respectively or all, or some of the units can be further split into a plurality of units with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions, and in actual application, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the image processing device can also include other units, and in actual application, these functions can also be realized by other units, and can be realized by multiple units in cooperation.

[0250] According to another embodiment of the present application, the image processing device shown in Figure 2 and Figure 4 may be constructed, and the image processing method of the embodiments of the present application can be implemented, by running a computer program (including program code) capable of performing each step involved in the corresponding method shown in Figure 6 on a general computing device such as a computer including a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and the like processing and storage elements. The computer program can be recorded on a computer readable recording medium, for example, and loaded into the above-mentioned computing device through the computer readable recording medium, and run therein.

[0251] Based on the same inventive concept, the image processing apparatus provided in the embodiments of the present application has similar principles and beneficial effects to those of the image processing apparatus in the method embodiments of the present application. For brevity, the principles and beneficial effects of the method embodiments are not described here again.

[0252] Please refer to Figure 7 , Figure 7 A structural schematic diagram of an intelligent device provided in the embodiments of the present application, which includes at least a processor 701, a communication interface 702 and a memory 703. The processor 701, the communication interface 702 and the memory 703 can be connected through a bus or other means. The processor 701 (or Central Processing Unit, CPU) is the computing core and control core of the terminal, which can parse various instructions in the terminal and process various data of the terminal, for example, the CPU can be used to parse the power-on / off instruction sent by the user to the terminal and control the terminal to perform power-on / off operation; for another example, the CPU can transmit various interactive data between the internal structures of the terminal, and the like. The communication interface 702 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be controlled by the processor 701 to receive and transmit data; the communication interface 702 can also be used for transmission and interaction of internal data of the terminal. The memory 703 is a memory device in the terminal, which is used to store programs and data. It can be understood that the memory 703 here can include the built-in memory of the terminal, and of course can also include the expansion memory supported by the terminal. The memory 703 provides a storage space, which stores the operating system of the terminal, which can include but is not limited to: Android system, iOS system, Windows Phone system, etc., and the present application does not make any limitation on this.

[0253] In the embodiments of the present application, the processor 701 executes the executable program code in the memory 703 to perform the following operations:

[0254] Obtain a target image through the communication interface 702, and perform feature extraction on the target image to obtain attribute information of N objects contained in the target image, N being a positive integer;

[0255] Determine M preselected frames of the target image according to the attribute information of the N objects, wherein the number of objects in the jth preselected frame is less than the number of objects in the (j+1)th preselected frame, j, M being positive integers, and j is less than M, and M is less than or equal to N;

[0256] According to the M preselected boxes, a candidate position set of the target image is obtained, and a target cropping position is determined from the candidate position set according to attribute information of the N objects;

[0257] The target image is cropped by the target cropping position to obtain a cropped image.

[0258] As an optional embodiment, the attribute information of the N objects includes position information of key points corresponding to the N objects respectively; and the specific embodiment in which the processor 701 determines the M preselected boxes of the target image according to the attribute information of the N objects is as follows:

[0259] According to the position information of the key points of the N objects, a distance between the key points of the N objects and a center point of the target image is determined, and the N objects are sorted according to the distance of each object to obtain an object sorting set;

[0260] Wherein, the distance between the key point of the i th object and the center point of the target image is less than or equal to the distance between the key point of the i+1 th object and the center point of the target image, i is a positive integer, and i is less than N;

[0261] According to the position information of the first P objects in the object sorting set, the j th preselected box of the target image is determined;

[0262] Wherein, the j th preselected box includes the first P objects in the image, and the ratio of the area of the j th preselected box to the area of the target image is greater than a first proportion threshold, P and j are positive integers, and P is less than or equal to N, and j is less than or equal to M.

[0263] As an optional embodiment, the specific embodiment in which the processor 701 obtains the candidate position set of the target image according to the M preselected boxes is as follows:

[0264] The target cropping scale is obtained through the communication interface 702, and the scale of the M preselected boxes is adjusted based on the target cropping scale to obtain M candidate boxes of the target image;

[0265] Each candidate box is slid in the target image to obtain a candidate position set of the target image;

[0266] Wherein, the candidate position corresponding to the target candidate box is the position of the target candidate box in the target image after each sliding.

[0267] As an optional embodiment, the specific embodiment in which the processor 701 adjusts the scale of the M preselected boxes based on the target cropping scale to obtain the M candidate boxes of the target image is as follows:

[0268] According to the position relationship between the k th preselected box and the N objects, an extended box corresponding to the k th preselected box is determined, k is a positive integer, and k is less than or equal to M;

[0269] determine a first inner border and a second inner border corresponding to the kth preselected frame according to a distance between the extension frame corresponding to the kth preselected frame and the edge of the target image, wherein an area of the first inner border is smaller than an area of the second inner border;

[0270] if the area of the target cropping scale is greater than the area of the first inner border and smaller than the area of the second inner border, adjust the width-height ratio of the kth preselected frame according to the width-height ratio of the target cropping scale to obtain the kth candidate frame.

[0271] As an optional embodiment, the image intercepted by the kth preselected frame includes Q objects, Q is a positive integer, and Q is smaller than N; the specific embodiment in which the processor 701 determines the first inner border and the second inner border corresponding to the kth preselected frame according to the distance between the kth preselected frame and the edge of the target image is as follows:

[0272] extend each side of the kth preselected frame to obtain an extension frame corresponding to the kth preselected frame;

[0273] wherein the ratio of the area of the image intercepted by the extension frame corresponding to the kth preselected frame that overlaps with the Q+1th object to the area of the Q+1th object is less than or equal to a second proportion threshold.

[0274] As an optional embodiment, the processor 701 further performs the following operations by running the executable program code in the memory 703:

[0275] if the area of the target cropping scale is less than or equal to the area of the first inner border, adjust the width-height ratio of the kth preselected frame according to the width-height ratio of the first inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame;

[0276] if the area of the target cropping scale is greater than or equal to the area of the second inner border, adjust the width-height ratio of the kth preselected frame according to the width-height ratio of the second inner border, and perform adaptive processing on the kth preselected frame according to the target cropping scale to obtain the kth candidate frame;

[0277] wherein the scale of the kth candidate frame matches the target cropping scale.

[0278] As an optional embodiment, the attribute information of the N objects includes the weight corresponding to each object and the coordinates of the key points of each object; the specific embodiment in which the processor 701 determines the target cropping position from the candidate position set according to the attribute information of the N objects is as follows:

[0279] calculate a first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, x is a positive integer and x is less than or equal to M;

[0280] According to the weight of each object in the xth candidate position and the coordinates of the key points of each object, a second score of the xth candidate position is calculated;

[0281] The first score and the second score are summed to obtain a cumulative score of the xth candidate position;

[0282] The candidate positions in the candidate position set whose cumulative scores are higher than the score threshold are determined as the target cropping positions.

[0283] As an optional embodiment, the processor 701 calculates the first score of the xth candidate position according to the preselected frame corresponding to the xth candidate position, and the specific embodiment is as follows:

[0284] According to the area relationship between the target cropping scale and the first inner frame and the second inner frame corresponding to the preselected frame corresponding to the xth candidate position, a first area score of the xth candidate position is calculated;

[0285] The ratio of the area of the xth candidate position to the area of the target image is calculated to obtain a second area score of the xth candidate position;

[0286] The first area score, the second area score, and the area of the target image are processed to obtain the first score of the xth candidate position.

[0287] As an optional embodiment, the xth candidate position and at least one object have an overlapping area, and the target object is any one of the at least one object; the processor 701 calculates the second score of the xth candidate position according to the weight of each object in the xth candidate position and the coordinates of the key points of each object, and the specific embodiment is as follows:

[0288] According to the overlapping area of the target object and the xth candidate position and the area of the target object, an overlapping area ratio is calculated, and according to the overlapping area ratio and the weight of the target object, an object score of the target object is calculated;

[0289] According to the coordinates of the key points of the target object, a key point score of the target object is calculated;

[0290] The object scores and the key point scores of the at least one object are accumulated to obtain the second score of the xth candidate position.

[0291] As an optional embodiment, the target image belongs to a target video; the processor 701 further performs the following operations by running the executable program code in the memory 703:

[0292] If the target image is a front y-frame image of the target video, the target image is subjected to edge detection, and y is a positive integer;

[0293] In response to the text object existing in the edge region of the target image, a third inner frame corresponding to the kth preselected frame is determined, and the text object is included in the third inner frame;

[0294] If the area of the target cropping scale is greater than the area of the first inner frame and less than the area of the second inner frame, the aspect ratio of the kth preselected frame is adjusted according to the aspect ratio of the third inner frame, and the kth preselected frame is adapted according to the target cropping scale to obtain the kth candidate frame.

[0295] The scale of the kth candidate frame matches the target cropping scale.

[0296] Based on the same inventive concept, the principles and beneficial effects of the intelligent device provided in the embodiments of the present application for solving problems are similar to those of the image processing method provided in the method embodiments of the present application for solving problems. For brevity, the principles and beneficial effects of the method are not described again here.

[0297] The embodiments of the present application also provide a computer readable storage medium, which stores one or more instructions. The one or more instructions are adapted to be loaded by a processor and execute the image processing method described in the above method embodiments.

[0298] The embodiments of the present application also provide a computer program product containing instructions, which, when running on a computer, causes the computer to execute the image processing method described in the above method embodiments.

[0299] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method described above.

[0300] The steps in the embodiments of the present application can be adjusted in sequence, combined and deleted according to actual needs.

[0301] The modules in the device embodiments of the present application can be combined, divided and deleted according to actual needs.

[0302] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing related hardware, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0303] The above disclosure is only a preferred embodiment of the present application, and of course cannot limit the scope of the present application, and those skilled in the art can understand that all or part of the processes described above can be implemented, and equivalent changes made in accordance with the claims of the present application, still belong to the scope covered by the present application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a target image and extract features from the target image to obtain attribute information of N objects contained in the target image, where N is a positive integer; Based on the attribute information of the N objects, M preselection boxes are determined for the target image, wherein the number of objects in the j-th preselection box is less than the number of objects in the (j+1)-th preselection box, j and M are positive integers, and j is less than M, M is less than or equal to N; the j-th preselection box includes P objects, P is a positive integer, and P is less than N, and the (j+1)-th preselection box is constructed based on the fact that the j-th preselection box includes P objects, and at least one object from the N objects other than the P objects included in the j-th preselection box; Based on the M preselected boxes, a set of candidate positions for the target image is obtained, and the target cropping position is determined from the set of candidate positions based on the attribute information of the N objects. The target image is cropped at the target cropping location to obtain the cropped image. The attribute information of the N objects includes the position information of the key points corresponding to each of the N objects; determining the M preselection boxes of the target image based on the attribute information of the N objects includes: determining the line distance between the key points of the N objects and the center point of the target image based on the position information of the key points of the N objects, and sorting the N objects according to the line distance of each object to obtain an object sorting set; determining the j-th preselection box of the target image based on the position information of the first P objects in the object sorting set.

2. The method as described in claim 1, characterized in that, in, The distance between the key point of the i-th object and the center point of the target image is less than or equal to the distance between the key point of the (i+1)-th object and the center point of the target image, where i is a positive integer and i is less than N; The image captured by the j-th preselection box includes the first P objects, and the ratio of the area of ​​the j-th preselection box to the area of ​​the target image is greater than a first ratio threshold, where P and j are both positive integers, and P is less than or equal to N and j is less than or equal to M.

3. The method as described in claim 1, characterized in that, The step of obtaining the candidate location set of the target image based on the M preselected boxes includes: Obtain the target cropping scale, and adjust the scale of the M pre-selected boxes based on the target cropping scale to obtain M candidate boxes of the target image; The candidate boxes are slid across the target image to obtain a set of candidate positions in the target image; The candidate position corresponding to the target candidate box is the position of the target candidate box in the target image after each sliding motion.

4. The method as described in claim 3, characterized in that, The step of adjusting the scale of the M pre-selected boxes based on the target cropping scale to obtain M candidate boxes of the target image includes: Based on the positional relationship between the kth preselection box and the N objects, determine the expansion box corresponding to the kth preselection box, where k is a positive integer and k is less than or equal to M; Based on the distance between the extended frame corresponding to the kth preselection box and the edge of the target image, the first inner border and the second inner border corresponding to the kth preselection box are determined, and the area of ​​the first inner border is smaller than the area of ​​the second inner border. If the area of ​​the target cropping scale is greater than the area of ​​the first inner border and less than the area of ​​the second inner border, then the aspect ratio of the kth preselection box is adjusted according to the aspect ratio of the target cropping scale to obtain the kth candidate box.

5. The method as described in claim 4, characterized in that, The image captured by the k-th preselection box includes Q objects, where Q is a positive integer and Q is less than N; determining the first inner border and the second inner border corresponding to the k-th preselection box based on the distance between the k-th preselection box and the edge of the target image includes: Expand each side of the kth preselection box to obtain the expanded box corresponding to the kth preselection box; Wherein, the ratio of the area of ​​the image captured by the expanded box corresponding to the k-th preselected box that overlaps with the (Q+1)-th object to the area of ​​the (Q+1)-th object is less than or equal to the second ratio threshold.

6. The method as described in claim 4, characterized in that, The method further includes: If the area of ​​the target cropping scale is less than or equal to the area of ​​the first inner border, then the aspect ratio of the kth preselection box is adjusted according to the aspect ratio of the first inner border, and the kth preselection box is adapted according to the target cropping scale to obtain the kth candidate box. If the area of ​​the target cropping scale is greater than or equal to the area of ​​the second inner border, then the aspect ratio of the kth preselection box is adjusted according to the aspect ratio of the second inner border, and the kth preselection box is adapted according to the target cropping scale to obtain the kth candidate box. The scale of the k-th candidate box is matched with the target cropping scale.

7. The method as described in claim 4, characterized in that, The attribute information of the N objects includes the weight corresponding to each object and the coordinates of the key points of each object; determining the target clipping position from the candidate position set based on the attribute information of the N objects includes: Based on the preselection box corresponding to the x-th candidate position, calculate the first score of the x-th candidate position, where x is a positive integer and x is less than or equal to M; The second score of the x-th candidate position is calculated based on the weight of each object in the x-th candidate position and the coordinates of the key points of each object. The first score and the second score are summed to obtain the cumulative score of the x-th candidate position; Candidate positions in the candidate position set whose cumulative score is higher than the score threshold are identified as target clipping positions.

8. The method as described in claim 7, characterized in that, The step of calculating the first score of the xth candidate position based on the preselected box corresponding to the xth candidate position includes: Based on the relationship between the target cropping scale and the area of ​​the first inner border and the second inner border of the preselected box corresponding to the xth candidate position, calculate the first area score of the xth candidate position; Calculate the ratio of the area of ​​the x-th candidate position to the area of ​​the target image to obtain the second area fraction of the x-th candidate position; The first area fraction, the second area fraction, and the area of ​​the target image are processed to obtain the first score of the xth candidate position.

9. The method as described in claim 7, characterized in that, The x-th candidate position overlaps with at least one object, and the target object is any one of the at least one objects; the step of calculating the second score of the x-th candidate position based on the weights of each object in the x-th candidate position and the coordinates of the key points of each object includes: Based on the overlapping area of ​​the target object and the xth candidate position, and the area of ​​the target object, calculate the overlapping area ratio, and based on the overlapping area ratio and the weight of the target object, calculate the object score of the target object; Calculate the key point score of the target object based on the coordinates of its key points; The object score and keypoint score of the at least one object are summed to obtain the second score of the x-th candidate position.

10. The method as described in claim 4, characterized in that, The target image belongs to the target video; the method further includes: If the target image is the first y frames of the target video, then edge detection is performed on the target image, where y is a positive integer; In response to the presence of a text object in the edge region of the target image, a third inner border corresponding to the kth preselection box is determined, and the third inner border includes the text object. If the area of ​​the target cropping scale is greater than the area of ​​the first inner border and less than the area of ​​the second inner border, then the aspect ratio of the kth preselection box is adjusted according to the aspect ratio of the third inner border, and the kth preselection box is adapted according to the target cropping scale to obtain the kth candidate box. The scale of the k-th candidate box is matched with the target cropping scale.

11. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the target image; The processing unit is used to extract features from the target image to obtain attribute information of N objects contained in the target image, where N is a positive integer; The system is configured to: determine M preselection boxes for the target image based on the attribute information of the N objects, wherein the number of objects in the j-th preselection box is less than the number of objects in the (j+1)-th preselection box, j and M are positive integers, and j is less than M, M is less than or equal to N; the j-th preselection box includes P objects, P is a positive integer, and P is less than N; the (j+1)-th preselection box is constructed based on the j-th preselection box including P objects, and at least one object from the N objects other than the P objects included in the j-th preselection box; obtain a candidate position set for the target image based on the M preselection boxes, and determine the target cropping position from the candidate position set based on the attribute information of the N objects; and crop the target image using the target cropping position to obtain a cropped image. The attribute information of the N objects includes the location information of the key points corresponding to each of the N objects; the processing unit is used to determine the line distance between the key points of the N objects and the center point of the target image based on the location information of the key points of the N objects, and sort the N objects according to the line distance of each object to obtain an object sorting set; and determine the j-th preselection box of the target image based on the location information of the first P objects in the object sorting set.

12. A smart device, characterized in that, include: Storage devices and processors; The storage device stores a computer program; A processor that executes a computer program to implement the image processing method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1-10.

14. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, the image processing method as described in any one of claims 1-10 is implemented.

Images