Multimedia resource recommendation method, device, electronic device and storage medium
By extracting and analyzing the relationship between additional information areas and key areas from multimedia resources, the problem of inaccurate recognition of "text psoriasis" in traditional technology is solved, and the accurate recommendation of multimedia resources is achieved.
Patent Information
- Application Number
- CN202011157425.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-11-22
AI Technical Summary
In traditional technology, the impact of 'text psoriasis' on multimedia resources cannot be accurately identified through the proportion of text areas in the cover image, resulting in inaccurate recommendations of multimedia resources.
Obtain the target image from the multimedia resource, extract the additional information area and key area, generate a feature set based on the object structure relationship and spatial position relationship of the additional information area and key area, analyze and obtain visual scores, and then recommend multimedia resource.
By comprehensively considering the relationship between additional information areas and key areas, the accuracy of visual scores is improved and the accurate recommendation of multimedia resources is ensured.
Smart Images

Figure CN112256891B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for recommending multimedia resources. Background Art
[0002] With the rapid development of multimedia technology and computer networks, short video technology has emerged. Through short video technology, users can publish multimedia resources, and each multimedia resource has a corresponding cover picture. The quality of the cover picture is an important factor in attracting users to click on multimedia resources.
[0003] Regarding the quality issues of cover images, a relatively large category of problems is called "text psoriasis". For example, the text in the cover image blocks the key areas of the user (such as the face), or a large amount of text is added to the cover image. In traditional technology, the text area of the cover image is generally detected through a text detection algorithm, and the impact of "text psoriasis" on multimedia resources is identified based on the proportion of the text area in the cover image.
[0004] However, in traditional technologies, the impact of "text psoriasis" on multimedia resources cannot be accurately identified by the proportion of text area in the cover image, which leads to inaccurate recommendations for multimedia resources. Summary of the invention
[0005] The present disclosure provides a multimedia resource recommendation method, device, electronic device and storage medium, so as to at least solve the problem in the related art that the influence of "text psoriasis" on multimedia resources cannot be accurately identified by the proportion of text area in the cover image, thereby causing inaccurate recommendation of multimedia resources. The technical solution of the present disclosure is as follows:
[0006] According to a first aspect of an embodiment of the present disclosure, a method for recommending multimedia resources is provided, comprising:
[0007] Acquire target images from multimedia resources;
[0008] Extracting a target area from the target image, the target area including an additional information area and a key area, the additional information area being an area in the target image used to display additional information, and the key area being a focus area when browsing or viewing the target image or being an area in the target image displaying key information;
[0009] Generate a feature set of the target image according to first position information of the additional information region in the target image and second position information of the key region in the target image, wherein the features in the feature set are used to characterize the object structure relationship and spatial position relationship between the additional information region and the key region;
[0010] Analyzing the feature set of the target image to obtain a perception score of the target image, wherein the perception score is used to characterize the degree of influence of the additional information on the key area;
[0011] The multimedia resource is recommended according to the perception score of the target image.
[0012] In one embodiment, the target area further includes a main body area, and the main body area is an area in the target image used to display a main object; extracting the target area from the target image includes:
[0013] Performing text recognition processing on the target image to obtain the additional information area;
[0014] Performing image segmentation on the target image to obtain the main area and the key area;
[0015] The step of generating a feature set of the target image according to the first position information of the additional information area in the target image and the second position information of the key area in the target image comprises:
[0016] Based on the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main area in the target image, a feature set of the target image is generated, and the features in the feature set are used to characterize the object structure relationship and spatial position relationship of the additional information area, the key area, and the main area.
[0017] In one embodiment, the step of performing text recognition processing on the target image to obtain the additional information area includes:
[0018] Performing encoding and decoding processing on the target image to obtain a plurality of candidate detection frames;
[0019] Screening the plurality of candidate detection frames to obtain a plurality of target detection frames;
[0020] Extracting connected domains from the areas corresponding to the plurality of target detection frames, and obtaining a circumscribed rectangular frame of the connected domain;
[0021] The circumscribed rectangular frames of the connected domain are merged to obtain the additional information region.
[0022] In one embodiment, the step of performing image segmentation on the target image to obtain the main area and the key area includes:
[0023] Performing image segmentation on the target image to obtain high-level semantic features of a main area and the main area;
[0024] Focusing high-level semantic features of the main area through an attention model to determine the position of the key area in the main area;
[0025] The key area is extracted from the main area according to the position of the key area in the main area.
[0026] In one embodiment, generating a feature set of the target image according to first position information of the additional information area in the target image, second position information of the key area in the target image, and third position information of the main area in the target image includes:
[0027] Determining an object structure relationship among the additional information area, the key area, and the main area;
[0028] Divide the target image into grids according to a preset rule to obtain grid distribution information of the target image;
[0029] Determine a spatial position relationship among the additional information area, the key area, and the main body area according to the first position information, the second position information, the third position information, and the grid distribution information of the target image;
[0030] A feature set of the target image is generated according to the object structure relationship and the spatial position relationship.
[0031] In one embodiment, determining the spatial position relationship between the additional information area, the key area, and the main area according to the first position information, the second position information, the third position information, and the grid distribution information of the target image includes:
[0032] Determine the grid distribution information of each of the additional information area, the key area, and the main area according to the first position information, the second position information, the third position information, and the grid distribution information;
[0033] According to the grid distribution information of the additional information area, the key area and the main area, the intersection and union ratio of the additional information area to the main area, the intersection and union ratio of the additional information area to the key area and the intersection and union ratio of the key area to the main area are obtained.
[0034] In one embodiment, the feature set further includes the number of the additional information areas; and the step of analyzing the feature set of the target image to obtain the perception score of the target image includes:
[0035] Determining, according to the number of the additional information areas and the first position information, coverage of the target image by the additional information;
[0036] The perception score of the target image is determined according to the coverage of the target image by the additional information.
[0037] In one embodiment, analyzing the feature set of the target image to obtain the perception score of the target image includes:
[0038] A regression analysis is performed on the feature set of the target image to obtain a perception score of the target image.
[0039] According to a second aspect of an embodiment of the present disclosure, a multimedia resource recommendation device is provided, including:
[0040] A target image acquisition module is configured to acquire a target image from a multimedia resource, wherein the target image is an image obtained by synthesizing additional information into an original image;
[0041] a target area extraction module, configured to extract a target area from the target image, wherein the target area includes an additional information area and a key area, wherein the additional information area is an area in the target image used to display additional information, and the key area is a key area of interest when browsing or viewing the target image or an area in the target image that displays key information;
[0042] a feature set generating module, configured to generate a feature set of the target image according to first position information of the additional information area in the target image and second position information of the key area in the target image;
[0043] a feature set analysis module, configured to analyze the feature set of the target image to obtain a perception score of the target image, wherein the perception score is used to characterize the degree of influence of the additional information on the key area;
[0044] The multimedia resource recommendation module is configured to recommend the multimedia resource according to the perception score of the target image.
[0045] In one embodiment, the target area also includes a main body area, and the main body area is an area in the target image used to display the main object; the target area extraction module includes a text processing unit and an image segmentation unit;
[0046] The text processing unit is configured to perform text recognition processing on the target image to obtain the additional information area;
[0047] The image segmentation unit is configured to perform image segmentation on the target image to obtain the main area and the key area;
[0048] The feature set generation module is also configured to generate a feature set of the target image based on the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main area in the target image.
[0049] In one of the embodiments, the text processing unit is configured to perform encoding and decoding processing on the target image to obtain a number of candidate detection frames; screen the several candidate detection frames to obtain a number of target detection frames; extract connected domains for areas corresponding to the several target detection frames and obtain a circumscribed rectangular frame of the connected domain; merge the circumscribed rectangular frames of the connected domain to obtain the additional information area.
[0050] In one of the embodiments, the image segmentation unit is further configured to perform image segmentation on the target image to obtain high-level semantic features of the main area and the main area; focus on the high-level semantic features of the main area through an attention model to determine the position of the key area in the main area; and extract the key area from the main area according to the position of the key area in the main area.
[0051] In one of the embodiments, the feature set generation module is further configured to determine the object structure relationship between the additional information area, the key area and the main area; divide the target image into grids according to preset rules to obtain grid distribution information of the target image; determine the spatial position relationship between the additional information area, the key area and the main area based on the first position information, the second position information, the third position information and the grid distribution information of the target image; and generate a feature set of the target image based on the object structure relationship and the spatial position relationship.
[0052] In one of the embodiments, the feature set generation module is further configured to determine the grid distribution information of the additional information area, the key area and the main area according to the first position information, the second position information, the third position information and the grid distribution information; and obtain the intersection-and-union ratio of the additional information area and the main area, the intersection-and-union ratio of the additional information area and the key area, and the intersection-and-union ratio of the key area and the main area according to the grid distribution information of the additional information area, the key area and the main area.
[0053] In one of the embodiments, the feature set also includes the number of the additional information areas; the feature set analysis module is further configured to determine the coverage of the target image by the additional information based on the number of the additional information areas and the first position information; and determine the perception score of the target image based on the coverage of the target image by the additional information.
[0054] In one of the embodiments, the feature set analysis module is further configured to perform regression analysis on the feature set of the target image to obtain a perception score of the target image.
[0055] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:
[0056] processor;
[0057] a memory for storing instructions executable by the processor;
[0058] The processor is configured to execute the instructions to implement the multimedia resource recommendation method described in any one of the embodiments of the first aspect.
[0059] According to a fourth aspect of an embodiment of the present disclosure, a storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device / server to execute the multimedia resource recommendation method as described in any one of the embodiments of the first aspect.
[0060] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device executes the method for recommending multimedia resources described in any one of the embodiments of the first aspect.
[0061] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0062] By acquiring a target image from multimedia resources; extracting an additional information region and a key region from the target image; generating a feature set of the target image according to first position information of the additional information region in the target image and second position information of the key region in the target image; analyzing the feature set of the target image to obtain a perception score of the target image; and recommending the multimedia resource according to the perception score of the target image. The present disclosure comprehensively considers the object structure relationship and spatial position relationship between the additional information region and the key region, determines the influence of the additional information region on the key region, improves the accuracy of perception score prediction, and thus ensures accurate recommendation of multimedia resources.
[0063] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0065] Figure 1 The figure is a diagram showing an application environment of a multimedia resource recommendation method according to an exemplary embodiment.
[0066] Figure 2 The figure is a flowchart of a method for recommending multimedia resources according to an exemplary embodiment.
[0067] Figure 3 The figure is a flowchart of a method for recommending multimedia resources according to an exemplary embodiment.
[0068] Figure 4a is a flowchart of step S310 according to an exemplary embodiment.
[0069] Figure 4b The figure is a flowchart of extracting a foreground text area according to an exemplary embodiment.
[0070] Figure 5a is a flowchart of step S320 according to an exemplary embodiment.
[0071] Figure 5b The figure is a flowchart of extracting a main area and a key area according to an exemplary embodiment.
[0072] Figure 6 is a flowchart of step S330 according to an exemplary embodiment.
[0073] Figure 7is a flowchart of step S630 according to an exemplary embodiment.
[0074] Figure 8 is a flowchart of step S240 according to an exemplary embodiment.
[0075] Figure 9a The figure is a flowchart of a method for recommending multimedia resources according to an exemplary embodiment.
[0076] Figure 9b is a network structure diagram of a decision tree model according to an exemplary embodiment.
[0077] Fig.10 The figure is a block diagram of a multimedia resource recommendation device according to an exemplary embodiment.
[0078] Fig.11 The figure is a diagram showing the internal structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0079] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0081] The multimedia resource recommendation method provided by the present disclosure can be applied to Figure 1 In the application environment shown. It includes: a terminal 110, a first electronic device 120 and a second electronic device 130. The first electronic device 120 and the second electronic device 130 refer to electronic devices with strong data storage and computing capabilities. For example, the first electronic device 120 and the second electronic device 130 can be a PC (Personal Computer) or a server, and the first electronic device 120 and the second electronic device 130 can be implemented with an independent server or a server cluster composed of multiple servers. The terminal 110 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices.
[0082] Among them, the terminal 110 has at least one of a browser, a social application, a live broadcast application, a shopping application or a payment application installed and running. The first electronic device 120 pushes similar multimedia resources to the terminal 110 corresponding to the user account based on the user data such as the viewing history and search history corresponding to the user account. The display interface of the application installed on the terminal 110 (such as the same city page, recommendation page, discovery page, etc.) displays the multimedia resources pushed by the first electronic device 120. The second electronic device 130 can be used to construct a machine learning model such as a full convolution segmentation network, and the constructed machine learning model is trained by the second electronic device 130, and the trained machine learning model can be deployed on the first electronic device 120. It can be understood that the first electronic device 120 pushes multimedia resources to the terminal 110 by comprehensively considering factors such as user data and the visual quality of multimedia resources. The multimedia resource recommendation method involved in the present disclosure is mainly aimed at the visual quality of multimedia resources. The first electronic device 120 obtains a target image from a multimedia resource, where the target image is an image obtained by synthesizing additional information into an original image; extracts a target area from the target image, where the target area includes an additional information area and a key area, where the additional information area is an area in the target image used to display additional information, and the key area is a key area of focus when browsing or viewing the target image, or is an area in the target image that displays key information; generates a feature set of the target image based on first position information of the additional information area in the target image and second position information of the key area in the target image; analyzes the feature set of the target image to obtain a perception score of the target image, where the perception score is used to characterize the degree of influence of the additional information on the visual effect of the key area; and recommends multimedia resources based on the perception score of the target image.
[0083] Figure 2 is a flowchart of a method for recommending multimedia resources according to an exemplary embodiment. Figure 2 As shown, the multimedia resource recommendation method is used in the first electronic device 120, and includes the following steps:
[0084] In step S210, a target image is acquired from multimedia resources.
[0085] The multimedia resource may be at least one of a text resource, a video resource, an audio resource, a picture resource or a webpage resource. The original image may be a picture included in the multimedia resource, or may be a picture used to represent the multimedia resource, such as a cover picture of the multimedia resource, a key frame image of a video resource or a picture resource. The additional information may be information such as pictures, animations, texts, emoticons, etc. added to the original image. For example, the additional information is text description information, which is added to the original image. The added text description information may clearly express part of the information of the multimedia resource.
[0086] Specifically, the multimedia resource has a corresponding target image, and the target image can be used to represent the main information or summarized information carried by the multimedia resource. The target image can also be used to display the information that the author of the multimedia resource intends to inform the public. On the one hand, when processing the multimedia resource, the multimedia resource is obtained from a server locally on the first electronic device or connected to the network of the first electronic device, and then the target image is obtained from the multimedia resource. On the other hand, the target image is obtained from the multimedia resource in advance, the target image is saved locally on the first electronic device or on the server connected to the network of the first electronic device, and the target image is obtained when processing the multimedia resource.
[0087] In step S220, a target area is extracted from the target image.
[0088] Among them, the target area includes an additional information area and a key area. The additional information area is an area in the target image used to display additional information. The key area is an image area selected from the target image, and the key area can be the focus (such as the area of interest) that the user pays attention to when browsing or viewing the target image. For example, the target image is a photo of a person, and the face of the person in the photo can be the area of interest. The key area can also be an area in the target image that displays key information. For example, the target image is a photo of a model, and the target image mainly displays the clothes on the model, and the corresponding area of the clothes in the photo can be the key area. Specifically, the target image includes an additional information area and a key area. The additional information is the information that the author of the multimedia resource intends to inform the public, and the information displayed in the key area is the focus of the user. When processing multimedia resources, since the target image is an image obtained by adding additional information to the original image, it is necessary to consider whether the information displayed in the key area can be fully presented to the user in the target image, that is, whether the additional information has an impact on the display of the key area. Therefore, in order to analyze whether the additional information area has an impact on the key area, the additional information area and the key area can be extracted from the target image through a machine learning model. The machine learning model can be a fully convolutional segmentation network (Fully Convolutional Networks, FCN). Exemplarily, since the additional information area may cover the key area, or the additional information area may cover most of the target image, the visual quality of the target image will be reduced. Therefore, based on the bad feeling brought to the user by the additional information area, the additional information area can be called psoriasis. If the additional information is text information, the additional information area can be called text psoriasis, and the target image can be called a psoriasis picture. The machine learning model that extracts the additional information area and the key area from the target image can be called a psoriasis recognition model. It should be noted that in the actual process of solving the problem, there will be other types of psoriasis, such as emoticon psoriasis.
[0089] In step S230, a feature set of the target image is generated according to the first position information of the additional information region in the target image and the second position information of the key region in the target image.
[0090] The features in the feature set are used to characterize the object structure relationship and spatial position relationship between the additional information area and the key area. The feature set of the target image includes feature data of the additional information area, feature data of the key area, and the relationship between the additional information area and the key area. The feature data of the additional information area can be the visual semantics of the additional information area or the distribution information of the additional information area. The feature data of the key area can be the visual semantics of the key area or the distribution information of the key area.
[0091] Specifically, the target image is input into a first machine learning model, the additional information area is extracted from the target image by the first machine learning model, and the first machine learning model outputs the first position information of the additional information area in the target image. The target image is input into a second machine learning model, the key area is extracted from the target image by the second machine learning model, and the second machine learning model outputs the second position information of the key area in the target image. The distribution information of the additional information area can be determined according to the first position information of the additional information area in the target image. The distribution information of the key area can be determined according to the second position information of the key area in the target image. The feature set of the target image is constructed using the distribution information of the additional information area, the visual semantics of the additional information area, the distribution information of the key area, and the visual semantics of the key area. It can be understood that the first machine learning model and the second machine learning model can constitute a psoriasis recognition model.
[0092] In step S240, the feature set of the target image is analyzed to obtain a perception score of the target image.
[0093] Among them, the perception score is used to characterize the degree of influence of additional information on the visual effect of the key area. For example, the negative impact of additional information on the visual effect of the key area is judged by using the coverage area of the additional information area on the key area or the positional relationship between the additional information area and the key area. If the coverage area is larger, the negative impact of the additional information on the visual effect of the key area is greater, and the perception score is higher; if the coverage area is smaller, the negative impact of the additional information on the visual effect of the key area is smaller, and the perception score is lower. If the positional relationship between the additional information area and the key area is closer, the negative impact of the additional information on the visual effect of the key area is greater, and the perception score is higher; if the positional relationship between the additional information area and the key area is farther, the negative impact of the additional information on the visual effect of the key area is smaller, and the perception score is higher. It can be understood that the higher the perception score of the psoriasis picture, the more severe the psoriasis of the target image.
[0094] Specifically, the feature set of the target image may include distribution information of the additional information area, visual semantics of the additional information area, distribution information of the key area, and visual semantics of the key area. Iterative operations are performed using each element in the feature set, and the results of the iterative operations and each element in the feature set are combined for analysis to obtain the probability relationship between each element and the degree of influence of the additional information on the key area, and the probability relationship is expressed as a perception score. The higher the perception score, the greater the negative impact of the additional information on the visual effect of the key area.
[0095] In step S250, multimedia resources are recommended according to the perception score of the target image.
[0096] Specifically, the feature set of the target image is analyzed to obtain the perception score of the target image. Since the target image can represent multimedia resources, the multimedia resources are displayed to the user through the target image before the user opens the multimedia resources. It can be seen that the quality of the target image affects the user's perception index. Therefore, the first electronic device pushes multimedia resources to the terminal based on the perception score of the target image. Exemplarily, the terminal is installed with a short video application (such as a Kuaishou application), and the application has at least one of a follow page, a discovery page (i.e., a recommended page), and a same-city page. These pages can be the homepage or specific pages of the application. In order to improve the user's perception index, the exposure traffic of multimedia resources with higher perception scores is suppressed, that is, the appearance of multimedia resources with higher perception scores on at least one of the follow page, the discovery page, and the same-city page is reduced.
[0097] In traditional technology, the proportion of text area in the cover image cannot accurately identify the impact of "text psoriasis" on multimedia resources. Therefore, the perception score obtained based on the proportion of text area in the whole image is also inaccurate. For example, in short video scenes, only the proportion of text area cannot accurately describe whether the text area affects the perception of the cover image or video. For example, in a selfie-type short video, a line of text is added to the face of the character in the video image. The proportion of the added text area in the video image is very low. The perception score determined only by the proportion cannot truly reflect the impact of the text area on the visual effect of the video image, that is, it cannot accurately reflect the user's perception. Therefore, when traditional technology recommends multimedia resources based on the perception score of the cover image, there is a technical problem of inaccurate recommendation of multimedia resources due to the inaccurate perception score. In the above-mentioned method for recommending multimedia resources, a target image is obtained from the multimedia resources; an additional information area and a key area are extracted from the target image; a feature set of the target image is generated according to the first position information of the additional information area in the target image and the second position information of the key area in the target image; the feature set of the target image is analyzed to obtain a perception score of the target image; and multimedia resources are recommended according to the perception score of the target image. In this embodiment, the accuracy of the perception score is improved by comprehensively considering the object structure relationship and spatial position relationship between the additional information area and the key area, so that when recommending multimedia resources according to the perception score of the target image, the technical problem of inaccurate recommendation of multimedia resources is solved, thereby improving the perception quality of the target image displayed on the terminal display interface.
[0098] In an exemplary embodiment, the target area also includes a main body area, which is an area in the target image used to display the main object. Figure 3 As shown, in step S220, the target area is extracted from the target image, which can be specifically achieved by the following steps:
[0099] In step S310, text recognition processing is performed on the target image to obtain an additional information area.
[0100] Among them, the additional information is text information, and the additional information area includes but is not limited to text added by the user through a banner (a rectangular bulletin board across the web page), text added to the target image by the user through other tools, text that does not appear in the background of the target image, and text that does not appear in the main area. Specifically, in order to extract the additional information area from the target image, the target image is subjected to text recognition processing to identify all text information in the target image. The identified text information may include text carried by the original image itself, not text added to the original image by the user. Therefore, in order to ensure that an accurate additional information area is extracted, the identified text information is filtered to obtain the additional information added to the original image, that is, the identified additional information is obtained. Further, the additional information area is extracted from the target image, and the first position information of the additional information area in the target image is obtained.
[0101] In step S320, the target image is segmented to obtain a main area and a key area.
[0102] Among them, image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing targets of interest or key targets. Specifically, the target image includes a main area and a key area. For example, if the target image is a selfie of a person, the main area can be the human body and the key area can be the face. The target image can be segmented by an image segmentation model, and the main area and the key area can be extracted from the target image, and the second position information of the key area in the target image and the third position information of the main area in the target image can be obtained.
[0103] In step S230, a feature set of the target image is generated according to the first position information of the additional information area in the target image and the second position information of the key area in the target image. This can be achieved by the following steps:
[0104] In step S330, a feature set of the target image is generated according to the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main body area in the target image.
[0105] Among them, the features in the feature set are used to characterize the object structure relationship and spatial position relationship of the additional information area, the key area and the main body area. Specifically, the additional information area is extracted from the target image, and the first position information of the additional information area in the target image is obtained. The target image is segmented by an image segmentation model, and the main body area and the key area are extracted from the target image, and the second position information of the key area in the target image and the third position information of the main body area in the target image are obtained. The distribution information of the additional information area can be determined according to the first position information of the additional information area in the target image. The distribution information of the key area can be determined according to the second position information of the key area in the target image. The distribution information of the main body area can be determined according to the second position information of the main body area in the target image. The feature set of the target image is constructed by using the distribution information of the additional information area, the visual semantics of the additional information area, the distribution information of the key area, the visual semantics of the key area, the distribution information of the main body area and the visual semantics of the main body area.
[0106] In the above-mentioned multimedia resource recommendation method, by extracting additional information areas, main areas and key areas from the target image, and generating a feature set of the target image according to the position information of the additional information areas, main areas and key areas in the target image, the relationship between the main area and the additional information area, and the relationship between the main area and the key area can be obtained, thereby realizing the comprehensive construction of the target image feature set, providing a good data foundation for the accurate prediction of subsequent perception scores, thereby solving the technical problem of inaccurate multimedia resource recommendations, and further improving the perception quality of the target image displayed on the terminal display interface.
[0107] In an exemplary embodiment, if Figure 4a As shown, in step S310, text recognition processing is performed on the target image to obtain an additional information area, which can be specifically achieved by the following steps:
[0108] In step S410, the target image is encoded and decoded to obtain a number of candidate detection frames.
[0109] Specifically, the target image is input into a fully convolutional segmentation network, which includes an encoding part and a decoding part. The target image is encoded and decoded through the fully convolutional segmentation network to recognize the text in the target image and obtain several candidate detection boxes.
[0110] In step S420, several candidate detection frames are screened to obtain several target detection frames.
[0111] Specifically, the text in the additional information has different features from the text in the original image itself, such as different font sizes, so the area sizes of their respective detection frames are also different. Several candidate detection frames can be screened according to the area of the detection frame to obtain the target detection frame corresponding to the additional information text.
[0112] In step S430, connected domains are extracted from the areas corresponding to the plurality of target detection frames, and the circumscribed rectangular frames of the connected domains are obtained.
[0113] Specifically, the target detection frames corresponding to the additional information text are connected to obtain the corresponding connected domain, and the connected domain is extracted. Each connected domain corresponds to a circumscribed rectangular frame, so the circumscribed rectangular frame of the connected domain is obtained.
[0114] In step S440, the circumscribed rectangular frames of the connected domain are merged to obtain an additional information region.
[0115] Specifically, each additional information may correspond to at least one connected domain, and each connected domain corresponds to a circumscribed rectangular frame. Each circumscribed rectangular frame is subjected to NMS merging processing to obtain an additional information region.
[0116] Exemplarily, the additional information is foreground text in the target image, and the additional information area is the foreground text area. Figure 4b The flowchart of the model for extracting the foreground text area is shown as an example. The foreground text area extraction method is to use the FCN network. The target image is input into the FCN network, and the input target image is encoded and decoded to obtain the candidate foreground text area (i.e., the candidate salient area), and then the final foreground text area is obtained through post-processing processes such as connected domain extraction, calculation of the external rectangular box, and NMS merging.
[0117] In the above-mentioned multimedia resource recommendation method, a target image is encoded and decoded to obtain several candidate detection frames; several candidate detection frames are screened to obtain several target detection frames; connected domains are extracted from the areas corresponding to the several target detection frames, and the circumscribed rectangular frames of the connected domains are obtained; the circumscribed rectangular frames of the connected domains are merged to obtain the additional information regions. This embodiment can ensure that accurate additional information regions are extracted from the target image, improve the accuracy of the target image feature set, and thus accurately identify the coverage of the additional information region on the key region and accurately predict the perception score.
[0118] In an exemplary embodiment, if Figure 5a As shown, in step S320, the target image is segmented to obtain the main area and the key area, which can be specifically achieved by the following steps:
[0119] In step S510, the target image is segmented to obtain high-level semantic features of the main area and the main area.
[0120] Among them, the high-level semantic features of an image refer to what can be seen. For example, for a face picture, extracting low-level features can be understood as extracting the outline of the face, and the high-level semantic features are a face. It can be seen that the semantic information of high-level semantic features is relatively rich, but the target position is relatively rough. The higher the level of features, the stronger the high-level semantics and the stronger the resolution. Specifically, the target image is input into a fully convolutional segmentation network. The fully convolutional segmentation network includes an encoding part and a decoding part. The encoding part and the decoding part respectively include several levels of convolutional layers. The target image is encoded and decoded through the fully convolutional segmentation network to obtain the high-level semantic features of the main area and the main area.
[0121] In step S520, the high-level semantic features of the main area are focused by using the attention model to determine the position of the key area in the main area.
[0122] In step S530, the key area is extracted from the main body area according to the position of the key area in the main body area.
[0123] Among them, the attention model is generated by drawing on the visual attention mechanism, which is a brain signal processing mechanism unique to human vision. Human vision quickly scans the global image to obtain the target area that needs to be focused on, which is generally referred to as the focus of attention, and then invests more attention resources in this area to obtain more detailed information about the target that needs to be focused on, while suppressing other useless information. Specifically, the semantic information of high-level semantic features is relatively rich, but the target position is relatively rough. Therefore, based on the visual attention mechanism, the high-level semantic features of the main area are focused and the position of the key area in the main area is determined. Therefore, according to the position of the key area in the main area, the key area is extracted from the main area.
[0124] The key area can be the area of interest. Taking this as an example, the main area and the area of interest are extracted end-to-end by adding a visual attention mechanism on the basis of the FCN network. The attention mechanism mainly uses the characteristics of the main area to affect the extraction of the area of interest. Figure 5b The flowchart of the model extracting the main area and the area of interest is shown as an example. Furthermore, the introduction of the attention mechanism is in the training stage of the model. In the training stage, the area of interest can be marked (i.e., the location information bbox of the rectangular box of the visual area of interest), and the marked area of interest can be used to guide the learning of the model.
[0125] In the above multimedia resource recommendation method, the target image is segmented to obtain the high-level semantic features of the main area and the main area; the high-level semantic features of the main area are focused by the attention model to determine the position of the key area in the main area; the key area is extracted from the main area according to the position of the key area in the main area. The relationship between the main area and the key area can be obtained, providing more comprehensive and accurate data materials for constructing the feature set of the target image, analyzing the comprehensive and accurate feature set, obtaining an accurate perception score, and accurately evaluating the impact of the additional information area on the perception quality of the target image, providing a reliable reference for the subsequent suppression of exposure traffic of multimedia resources with poor perception quality.
[0126] In an exemplary embodiment, if Figure 6 As shown, in step S330, a feature set of the target image is generated according to the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main body area in the target image. Specifically, this can be achieved by the following steps:
[0127] In step S610, the object structure relationship among the additional information region, the key region and the main body region is determined.
[0128] Among them, the object structure relationship refers to the relationship between the object structures included in the target image. For example, if the target image is a selfie photo of a person, then the human body and face in the photo, and the text on the person's clothes belong to the same object structure, while the banner text added by the user belongs to another object structure. Specifically, the additional information area, key area and theme area are first extracted from the target image, and then when analyzing the perception quality of the target image, it is necessary to know the visual semantics of the additional information area, key area and theme area. Since the target image is an image obtained by synthesizing the additional information into the original image, the additional information belongs to one visual semantics, and the original image belongs to another visual semantics. It can be seen that in the target image, the additional information area and the original image belong to different object structures. The key area and theme area extracted from the target image belong to the same object structure as the original image, thereby determining the object structure relationship between the additional information area, the key area and the main area.
[0129] In step S620, the target image is divided into grids according to a preset rule to obtain grid distribution information of the target image.
[0130] Among them, the grid division divides the target image into several equal parts in the first direction and / or the second direction, and the first direction is perpendicular to the second direction. The preset rule may be a rule followed in dividing the target image. Specifically, the target image is divided into grids according to the preset rules to obtain the grid distribution information of the target image. For example, if the target image is divided into three equal parts in the length and width directions, the target image is divided into 9 rectangles to obtain the nine-grid distribution of the target image, which can be used to calculate the distribution of additional information areas, key areas, and main areas in the target image.
[0131] In step S630, the spatial position relationship among the additional information area, the key area and the main body area is determined according to the first position information, the second position information, the third position information and the grid distribution information of the target image.
[0132] Specifically, an additional information region is extracted from the target image, and first position information of the additional information region in the target image is obtained. The target image is segmented by an image segmentation model, and a main region and a key region are extracted from the target image, and second position information of the key region in the target image and third position information of the main region in the target image are obtained. According to the first position information, the second position information, the third position information and the grid distribution information of the target image, the distribution information of the additional information region, the key region and the main region in the target image is determined. According to the distribution information of the additional information region, the key region and the main region in the target image, the spatial position relationship between the additional information region, the key region and the main region is determined.
[0133] In step S640, a feature set of the target image is generated according to the object structure relationship and the spatial position relationship.
[0134] Specifically, under the premise that the object structure relationship and spatial position relationship between the additional information area, the key area and the main area are known, a feature set of the target image can be constructed based on the object structure relationship and spatial position relationship between various areas in the target image.
[0135] In the above-mentioned multimedia resource recommendation method, by determining the object structure relationship and spatial position relationship between the additional information area, the key area and the main area, a feature set of the target image is constructed according to the object structure relationship and spatial position relationship. The feature set constructed in this embodiment includes the object structure relationship and spatial position relationship between the additional information area, the key area and the main area, which can comprehensively reflect the relationship between each area, provide a comprehensive and accurate data basis for analyzing the perception quality of the target image, and improve the accuracy of perception score prediction.
[0136] In an exemplary embodiment, if Figure 7 As shown, in step S630, the spatial position relationship between the additional information area, the key area and the main area is determined according to the first position information, the second position information, the third position information and the grid distribution information of the target image, which can be specifically achieved by the following steps:
[0137] In step S710, the grid distribution information of each of the additional information area, the key area and the main area is determined according to the first position information, the second position information, the third position information and the grid distribution information.
[0138] Specifically, an additional information area is extracted from the target image, and first position information of the additional information area in the target image is obtained. The target image is segmented by an image segmentation model, and the main area and the key area are extracted from the target image, and second position information of the key area in the target image and third position information of the main area in the target image are obtained. The first position information is compared with the grid distribution information of the target image to determine the grid distribution information of the additional information area. The second position information is compared with the grid distribution information of the target image to determine the grid distribution information of the key area. The third position information is compared with the grid distribution information of the target image to determine the grid distribution information of the main area.
[0139] In step S720, the IoU of the additional information region to the main region, the IoU of the additional information region to the key region, and the IoU of the key region to the main region are obtained according to the grid distribution information of each of the additional information region, the key region, and the main region.
[0140] Among them, the intersection over union (IOU) is the ratio of the intersection and union of the areas of two rectangular boxes. Specifically, according to the grid distribution information of the additional information area and the grid distribution information of the main area, the intersection and union of the additional information area and the main area are obtained, and the intersection and union ratio of the additional information area and the main area is obtained according to the intersection and union of the additional information area and the main area. According to the grid distribution information of the additional information area and the grid distribution information of the key area, the intersection and union of the additional information area and the key area are obtained, and the intersection and union ratio of the additional information area and the key area is obtained according to the intersection and union of the additional information area and the key area. According to the grid distribution information of the key area and the grid distribution information of the main area, the intersection and union of the key area and the main area are obtained, and the intersection and union ratio of the key area and the main area is obtained according to the intersection and union of the key area and the main area.
[0141] In the above-mentioned multimedia resource recommendation method, based on the first position information, the second position information, the third position information and the grid distribution information, the grid distribution information of the additional information area, the key area and the main area is obtained, and then the intersection and union ratio of the additional information area and the main area, the intersection and union ratio of the additional information area and the key area, and the intersection and union ratio of the key area and the main area are determined. Through these features, the feature set of the target image is further enriched, and based on the feature set of the target image, the probability relationship between the three areas and the visual quality of the target image can be more accurately obtained, further improving the accuracy of the visual score.
[0142] In an exemplary embodiment, the feature set also includes the number of additional information areas. Figure 8 As shown, in step S240, the feature set of the target image is analyzed to obtain the perception score of the target image, which can be specifically achieved by the following steps:
[0143] In step S810, the coverage of the target image by the additional information is determined according to the number of additional information areas and the first position information.
[0144] Specifically, by extracting the additional information region from the target image, the first position information of the additional information region and the number of the additional information regions can be obtained. Thus, according to the number of the additional information regions and the first position information of the additional information regions, the proportion of the additional information regions in the target image can be obtained, that is, the coverage of the additional information on the target image can be determined.
[0145] In step S820, the perception score of the target image is determined according to the coverage of the target image by the additional information.
[0146] Specifically, the coverage of the target image by the additional information affects the visual quality of the target image. Therefore, the coverage of the target image by the additional information affects the visual score of the target image. The visual score of the target image can be determined according to the coverage of the target image by the additional information. For example, if there are multiple additional information areas in the target image, the proportion of the additional information area in the target image is determined according to the first position information of each additional information area. If the proportion is large, the additional information covers most of the target image, which will seriously affect the visual effect of the target image, so that the visual score of the target image is high.
[0147] In this embodiment, the coverage of the target image by the additional information is determined according to the number of additional information areas and the first position information. The perception score of the target image is determined according to the coverage of the target image by the additional information. Starting from the number of additional information areas and the position information of each additional information area, the factors that may affect the visual effect of the target image are comprehensively considered to determine the comprehensiveness of the target image feature set, thereby providing a data basis for the prediction of the perception score.
[0148] Figure 9a is a flowchart of a method for recommending multimedia resources according to an exemplary embodiment. Figure 9a As shown, the multimedia resource recommendation method is used in the first electronic device 120, and includes the following steps:
[0149] In step S902, a target image is acquired from multimedia resources.
[0150] In step S904, text recognition is performed on the target image to obtain an additional information area.
[0151] Specifically, the target image is encoded and decoded to obtain several candidate detection frames; the several candidate detection frames are screened to obtain several target detection frames; the connected domains are extracted from the areas corresponding to the several target detection frames, and the circumscribed rectangular frames of the connected domains are obtained; the circumscribed rectangular frames of the connected domains are merged to obtain additional information areas.
[0152] In step S906, the target image is segmented to obtain a main area and a key area.
[0153] Specifically, the target image is segmented to obtain high-level semantic features of the main area and the main area; the high-level semantic features of the main area are focused through an attention model to determine the position of the key area in the main area; and the key area is extracted from the main area according to the position of the key area in the main area.
[0154] In step S908, the object structure relationship among the additional information area, the key area and the main body area is determined.
[0155] In step S910, the target image is divided into grids according to a preset rule to obtain grid distribution information of the target image.
[0156] In step S912, the spatial position relationship among the additional information area, the key area and the main body area is determined according to the first position information, the second position information, the third position information and the grid distribution information of the target image.
[0157] Specifically, according to the first position information, the second position information, the third position information and the grid distribution information, the grid distribution information of the additional information area, the key area and the main area is determined; according to the grid distribution information of the additional information area, the key area and the main area, the intersection-over-combination ratio of the additional information area and the main area, the intersection-over-combination ratio of the additional information area and the key area, and the intersection-over-combination ratio of the key area and the main area are obtained.
[0158] In step S914, a feature set of the target image is generated according to the object structure relationship and the spatial position relationship.
[0159] In step S916, a regression analysis is performed on the feature set of the target image to obtain a perception score of the target image.
[0160] Specifically, a decision tree model is used to perform regression analysis on the feature set of the target image to obtain a perception score of the target image. Figure 9b The network structure diagram of the decision tree model is shown. Furthermore, the feature set also includes the number of additional information areas; the coverage of the target image by the additional information can be determined according to the number of additional information areas and the first position information; and the perception score of the target image can be determined according to the coverage of the target image by the additional information.
[0161] In step S918, multimedia resources are recommended according to the perception score of the target image.
[0162] Exemplarily, the multimedia resource may be a short video, the target image may be the cover of the short video, the additional information is the text added to the cover of the short video, and the text added to the cover is called text psoriasis. The additional information area is the foreground text area, the key area is the visual key area, and the main area is the cover content main area. First, the foreground text area, the cover content main area, and the visual key area are extracted through the text area detection algorithm, the main area detection algorithm, and the visual key area detection algorithm; then, the feature abstraction module is used to calculate and obtain the visual semantics and spatial position relationship between the three feature areas, and a feature set is generated based on the obtained visual semantics and spatial position relationship; finally, the perception score of the text psoriasis is predicted through the decision tree model. Figure 5bThe block diagram of the short video cover text psoriasis recognition system is also shown. Therefore, the recognition system mainly includes a feature region extraction module, a feature abstraction module and a text psoriasis perception estimation model based on a decision tree. Among them, the feature region extraction module is configured to perform foreground text region extraction, main body region extraction, and visual key region extraction. The feature abstraction module is configured to perform the calculation of the visual semantics and spatial position relationship between the feature regions, and construct a feature set based on the calculated visual speech and spatial position relationship. The text psoriasis perception estimation model based on the decision tree is configured to perform the prediction of the perception score of the text psoriasis based on the constructed feature set.
[0163] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0164] Fig.10 is a block diagram of a multimedia resource recommendation device according to an exemplary embodiment. Fig.10 The device includes a target image acquisition module 1002, a target area extraction module 1004, a feature set generation module 1006, a feature set analysis module 1008 and a multimedia resource recommendation module 1010.
[0165] The target image acquisition module 1002 is configured to acquire the target image from the multimedia resource;
[0166] The target region extraction module 1004 is configured to extract a target region from the target image, wherein the target region includes an additional information region and a key region, wherein the additional information region is a region in the target image used to display additional information, and the key region is a key region that is focused on when browsing or viewing the target image or is a region in the target image that displays key information;
[0167] A feature set generating module 1006 is configured to generate a feature set of the target image according to first position information of the additional information region in the target image and second position information of the key region in the target image, wherein the features in the feature set are used to characterize the object structure relationship and spatial position relationship between the additional information region and the key region;
[0168] The feature set analysis module 1008 is configured to analyze the feature set of the target image to obtain a perception score of the target image, where the perception score is used to represent the degree of influence of the additional information on the key area;
[0169] The multimedia resource recommendation module 1010 is configured to recommend the multimedia resource according to the perception score of the target image.
[0170] In an exemplary embodiment, the target area also includes a main body area, and the main body area is an area in the target image used to display the main object. The target area extraction module 1004 includes a text processing unit and an image segmentation unit; wherein:
[0171] The text processing unit is configured to perform text recognition processing on the target image to obtain the additional information area;
[0172] The image segmentation unit is configured to perform image segmentation on the target image to obtain the main area and the key area;
[0173] The feature set generation module 1006 is further configured to generate a feature set of the target image based on the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main area in the target image, wherein the features in the feature set are used to characterize the object structure relationship and spatial position relationship among the additional information area, the key area, and the main area.
[0174] In an exemplary embodiment, the text processing unit is configured to perform encoding and decoding processing on the target image to obtain a number of candidate detection frames; screen the several candidate detection frames to obtain a number of target detection frames; extract connected domains for areas corresponding to the several target detection frames and obtain a circumscribed rectangular frame of the connected domain; merge the circumscribed rectangular frames of the connected domain to obtain the additional information area.
[0175] In an exemplary embodiment, the image segmentation unit is further configured to perform image segmentation on the target image to obtain high-level semantic features of the main area and the main area; focus on the high-level semantic features of the main area through an attention model to determine the position of the key area in the main area; and extract the key area from the main area according to the position of the key area in the main area.
[0176] In an exemplary embodiment, the feature set generation module 1006 is further configured to determine the object structure relationship between the additional information area, the key area and the main area; divide the target image into grids according to preset rules to obtain grid distribution information of the target image; determine the spatial position relationship between the additional information area, the key area and the main area based on the first position information, the second position information, the third position information and the grid distribution information of the target image; and generate a feature set of the target image based on the object structure relationship and the spatial position relationship.
[0177] In an exemplary embodiment, the feature set generation module 1006 is further configured to determine the grid distribution information of the additional information area, the key area and the main area according to the first position information, the second position information, the third position information and the grid distribution information; and obtain the intersection and union ratio of the additional information area and the main area, the intersection and union ratio of the additional information area and the key area, and the intersection and union ratio of the key area and the main area according to the grid distribution information of the additional information area, the key area and the main area.
[0178] In an exemplary embodiment, the feature set also includes the number of the additional information areas; the feature set analysis module 1008 is further configured to determine the coverage of the target image by the additional information based on the number of the additional information areas and the first position information; and determine the perception score of the target image based on the coverage of the target image by the additional information.
[0179] In an exemplary embodiment, the feature set analysis module 1008 is further configured to perform regression analysis on the feature set of the target image to obtain a perception score of the target image.
[0180] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0181] Fig.11 1 is a block diagram of a device 1100 for executing a multimedia resource processing method according to an exemplary embodiment. For example, the device 1100 may be a server. Fig.11The device 1100 includes a processing component 1120, which further includes one or more processors, and a memory resource represented by a memory 1122 for storing instructions executable by the processing component 1120, such as an application. The application stored in the memory 1122 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1120 is configured to execute instructions to perform the above-mentioned multimedia resource recommendation method.
[0182] The device 1100 may also include a power component 1124 configured to perform power management of the device 1100, a wired or wireless network interface 1126 configured to connect the device 1100 to a network, and an input / output (I / O) interface 1128. The device 1100 may operate based on an operating system stored in the memory 1122, such as Windows, Mac OS X, Unix, Linux, FreeBSD, or the like.
[0183] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 1122 including instructions, which can be executed by a processor of the device 1100 to perform the above method. The storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0184] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0185] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for recommending multimedia resources, characterized in that: include: Acquire target images from multimedia resources; Extracting a target area from the target image, the target area including an additional information area and a key area, the additional information area being an area in the target image used to display additional information, and the key area being a focus area when browsing or viewing the target image or being an area in the target image displaying key information; Generate a feature set of the target image according to first position information of the additional information region in the target image and second position information of the key region in the target image, wherein the features in the feature set are used to characterize the object structure relationship and spatial position relationship between the additional information region and the key region, and the feature set includes visual semantics of the additional information region and visual semantics of the key region; Analyzing the feature set of the target image to obtain a perception score of the target image, wherein the perception score is used to characterize the degree of influence of the additional information on the key area; The multimedia resource is recommended according to the perception score of the target image.
2. The method for recommending multimedia resources according to claim 1, characterized in that: The target area also includes a main area, and the main area is an area in the target image used to display a main object; extracting the target area from the target image includes: Performing text recognition processing on the target image to obtain the additional information area; Performing image segmentation on the target image to obtain the main area and the key area; The step of generating a feature set of the target image according to the first position information of the additional information area in the target image and the second position information of the key area in the target image comprises: Based on the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main area in the target image, a feature set of the target image is generated, and the features in the feature set are used to characterize the object structure relationship and spatial position relationship of the additional information area, the key area, and the main area.
3. The method for recommending multimedia resources according to claim 2, characterized in that: The performing text recognition processing on the target image to obtain the additional information area includes: Performing encoding and decoding processing on the target image to obtain a plurality of candidate detection frames; Screening the plurality of candidate detection frames to obtain a plurality of target detection frames; Extracting connected domains from the areas corresponding to the plurality of target detection frames, and obtaining a circumscribed rectangular frame of the connected domain; The circumscribed rectangular frames of the connected domain are merged to obtain the additional information region.
4. The method for recommending multimedia resources according to claim 2, characterized in that: The step of performing image segmentation on the target image to obtain the main area and the key area includes: Performing image segmentation on the target image to obtain high-level semantic features of a main area and the main area; Focusing high-level semantic features of the main area through an attention model to determine the position of the key area in the main area; The key area is extracted from the main area according to the position of the key area in the main area.
5. The method for recommending multimedia resources according to claim 2, characterized in that: The step of generating a feature set of the target image according to the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main body area in the target image comprises: Determining an object structure relationship among the additional information area, the key area, and the main area; Divide the target image into grids according to a preset rule to obtain grid distribution information of the target image; Determine a spatial position relationship among the additional information area, the key area, and the main body area according to the first position information, the second position information, the third position information, and the grid distribution information of the target image; A feature set of the target image is generated according to the object structure relationship and the spatial position relationship.
6. The method for recommending multimedia resources according to claim 5, characterized in that: The determining, according to the first position information, the second position information, the third position information and the grid distribution information of the target image, the spatial position relationship among the additional information area, the key area and the main body area comprises: Determine the grid distribution information of each of the additional information area, the key area, and the main area according to the first position information, the second position information, the third position information, and the grid distribution information; According to the grid distribution information of the additional information area, the key area and the main area, the intersection and union ratio of the additional information area to the main area, the intersection and union ratio of the additional information area to the key area and the intersection and union ratio of the key area to the main area are obtained.
7. The method for recommending multimedia resources according to any one of claims 1 to 6, characterized in that: The feature set also includes the number of the additional information areas; and the analyzing the feature set of the target image to obtain the perception score of the target image includes: Determining, according to the number of the additional information areas and the first position information, coverage of the target image by the additional information; The perception score of the target image is determined according to the coverage of the target image by the additional information.
8. The method for recommending multimedia resources according to any one of claims 1 to 6, characterized in that: The step of analyzing the feature set of the target image to obtain a perception score of the target image includes: A regression analysis is performed on the feature set of the target image to obtain a perception score of the target image.
9. A multimedia resource recommendation device, characterized in that: include: A target image acquisition module is configured to acquire a target image from a multimedia resource; a target area extraction module, configured to extract a target area from the target image, wherein the target area includes an additional information area and a key area, wherein the additional information area is an area in the target image used to display additional information, and the key area is a key area of interest when browsing or viewing the target image or an area in the target image that displays key information; a feature set generating module, configured to generate a feature set of the target image according to first position information of the additional information region in the target image and second position information of the key region in the target image, wherein the features in the feature set are used to characterize the object structure relationship and spatial position relationship between the additional information region and the key region, and the feature set includes visual semantics of the additional information region and visual semantics of the key region; a feature set analysis module, configured to analyze the feature set of the target image to obtain a perception score of the target image, wherein the perception score is used to characterize the degree of influence of the additional information on the key area; The multimedia resource recommendation module is configured to recommend the multimedia resource according to the perception score of the target image.
10. The multimedia resource recommendation device according to claim 9, characterized in that: The target area also includes a main body area, which is an area in the target image used to display the main object; the target area extraction module includes a text processing unit and an image segmentation unit; The text processing unit is configured to perform text recognition processing on the target image to obtain the additional information area; The image segmentation unit is configured to perform image segmentation on the target image to obtain the main area and the key area; The feature set generation module is further configured to generate a feature set of the target image based on the first position information of the additional information area in the target image, the second position information of the key area in the target image, and the third position information of the main area in the target image, wherein the features in the feature set are used to characterize the object structure relationship and spatial position relationship among the additional information area, the key area, and the main area.
11. The multimedia resource recommendation device according to claim 10, characterized in that: The text processing unit is configured to perform encoding and decoding processing on the target image to obtain a plurality of candidate detection frames; screen the plurality of candidate detection frames to obtain a plurality of target detection frames; extract connected domains from areas corresponding to the plurality of target detection frames, and obtain a circumscribed rectangular frame of the connected domain; The circumscribed rectangular frames of the connected domain are merged to obtain the additional information region.
12. The multimedia resource recommendation device according to claim 10, characterized in that: The image segmentation unit is further configured to perform image segmentation on the target image to obtain high-level semantic features of the main area and the main area; focus the high-level semantic features of the main area through an attention model to determine the position of the key area in the main area; and extract the key area from the main area according to the position of the key area in the main area.
13. The multimedia resource recommendation device according to claim 10, characterized in that: The feature set generation module is further configured to determine the object structure relationship between the additional information area, the key area, and the main body area; divide the target image into grids according to a preset rule to obtain grid distribution information of the target image; determine the spatial position relationship between the additional information area, the key area, and the main body area according to the first position information, the second position information, the third position information, and the grid distribution information of the target image; A feature set of the target image is generated according to the object structure relationship and the spatial position relationship.
14. The multimedia resource recommendation device according to claim 13, characterized in that: The feature set generation module is further configured to determine the grid distribution information of the additional information area, the key area and the main area according to the first position information, the second position information, the third position information and the grid distribution information; and obtain the intersection and union ratio of the additional information area and the main area, the intersection and union ratio of the additional information area and the key area, and the intersection and union ratio of the key area and the main area according to the grid distribution information of the additional information area, the key area and the main area.
15. The multimedia resource recommendation device according to any one of claims 9 to 14, characterized in that: The feature set also includes the number of the additional information areas; the feature set analysis module is further configured to determine the coverage of the target image by the additional information based on the number of the additional information areas and the first position information; and determine the perception score of the target image based on the coverage of the target image by the additional information.
16. The multimedia resource recommendation device according to any one of claims 9 to 14, characterized in that: The feature set analysis module is further configured to perform regression analysis on the feature set of the target image to obtain a perception score of the target image.
17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the multimedia resource recommendation method according to any one of claims 1 to 8.
18. A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for recommending multimedia resources as claimed in any one of claims 1 to 8.
19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for recommending multimedia resources according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Image detection method comprising portrait in document and system
CN102496025A
Method and device for calculating psoriasis value of product image
CN104835134A
Image identification method, device and system
CN107545271A
Image processing method and device, storage medium and mobile terminal
CN108494996A
Image processing method and apparatus, storage medium, and mobile terminal
CN108765380A
