Multimedia information processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a multimedia information processing method and device, electronic equipment and a storage medium, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and auxiliary driving, and the method comprises the following steps: firstly obtaining multimedia information to be pushed, detecting the multimedia information to obtain promotion elements contained in the multimedia information, and sending the promotion elements to a server; then, the relevancy between the promotion elements and the multimedia information is calculated, saliency feature information of the promotion elements in the multimedia information is obtained, the saliency of the promotion elements in the multimedia information is calculated according to the saliency feature information, and therefore the pushing quality of the multimedia information is calculated according to the relevancy and the saliency. And pushing the multimedia information according to the pushing quality of the multimedia information. According to the technical scheme, the accuracy of the pushing quality of the multimedia information is improved, and the pushing effect is improved.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular, to a method and apparatus for processing multimedia information, an electronic device, a storage medium, and a program product. Background Art

[0002] With the rapid development of digital media technology, multimedia information such as advertisements can be pushed to users through digital media.

[0003] In the related art, before pushing multimedia information, usually only the multimedia information is evaluated according to whether the multimedia information violates relevant regulations and whether the size of the multimedia information matches the size of the display position, resulting in a poor pushing effect of the multimedia information. Summary of the Invention

[0004] Embodiments of the present application provide a method and apparatus for processing multimedia information, an electronic device, a storage medium, and a program product, which can detect the pushing quality of multimedia information, improve the accuracy of the pushing quality of multimedia information, and thus improve the pushing effect.

[0005] According to one aspect of the embodiments of the present application, a method for processing multimedia information is provided. The method includes:

[0006] Obtain the multimedia information to be pushed, and detect the multimedia information to obtain the promotion elements included in the multimedia information;

[0007] Calculate the correlation between the promotion elements and the multimedia information;

[0008] Obtain the salient feature information of the promotion elements in the multimedia information, and calculate the salience of the promotion elements in the multimedia information according to the salient feature information;

[0009] Calculate the pushing quality of the multimedia information according to the correlation and the salience, and push the multimedia information according to the pushing quality of the multimedia information.

[0010] According to one aspect of the embodiments of the present application, a multimedia information processing apparatus is provided. The apparatus includes:

[0011] A detection module configured to obtain the multimedia information to be pushed and detect the multimedia information to obtain the promotion elements included in the multimedia information;

[0012] A calculation module configured to calculate the correlation between the promotion elements and the multimedia information;

[0013] An acquisition module, configured to acquire the saliency feature information of the promotion element in the multimedia information, and calculate the saliency of the promotion element in the multimedia information according to the saliency feature information;

[0014] A push module, configured to calculate the push quality of the multimedia information according to the relevance and the saliency, and push the multimedia information according to the push quality of the multimedia information.

[0015] In an exemplary embodiment, based on the foregoing solution, the calculation module is specifically configured to: if multiple categories of promotion elements are detected, calculate the relevance between each category of promotion element and the multimedia information according to the identification information corresponding to each category of promotion element and the theme description information of the multimedia information;

[0016] The acquisition module is specifically configured to: acquire the saliency feature information of multiple feature categories of each category of promotion element in the multimedia information; calculate the initial saliency corresponding to the saliency feature information of each feature category; perform a summation operation on the initial saliencies corresponding to the saliency feature information of the multiple feature categories to obtain the saliency of each category of promotion element in the multimedia information.

[0017] In an exemplary embodiment, based on the foregoing solution, the processing module is specifically configured to: calculate the importance of each category of promotion element in the multimedia information according to the relevance between each category of promotion element and the multimedia information, and the saliency of each category of promotion element in the multimedia information; wherein, the importance is positively correlated with the relevance corresponding to each category of promotion element, and is positively correlated with the saliency of each category of promotion element in the multimedia information; perform a summation operation on the importances corresponding to the multiple categories of promotion elements to obtain the push quality of the multimedia information.

[0018] In an exemplary embodiment, based on the foregoing solution, the acquisition module is specifically configured to: search for the weights of the multiple feature categories corresponding to the category of the multimedia information from a set weight dataset; perform a summation operation on the initial saliencies corresponding to the saliency feature information of the multiple feature categories according to the weights of the multiple feature categories to obtain the saliency of each category of promotion element in the multimedia information.

[0019] In an exemplary embodiment, based on the foregoing solution, the obtaining module is specifically configured as follows: If the multimedia information is an image, obtain the number of elements of each type of promotion element included in the image, and find the central promotion element located at the center of the image from each type of promotion element; calculate the total area of each type of promotion element in the image, and obtain the color difference value between each type of promotion element and the background of the image.

[0020] In an exemplary embodiment, based on the foregoing solution, the obtaining module is specifically configured as follows: If there are multiple promotion elements included in each type of promotion element, find the area to which each promotion element belongs and other areas except the area to which each promotion element belongs from the rectangular detection frames of each promotion element included in the multiple promotion elements; calculate the average value of the color values of multiple pixels included in the area to which each promotion element belongs to obtain a first color value, and calculate the average value of the color values of multiple pixels included in the other areas to obtain a second color value; calculate the difference between the first color value and the second color value to obtain the color difference value corresponding to each promotion element, and calculate the color difference value between each type of promotion element and the background of the image according to the color difference values respectively corresponding to the multiple promotion elements.

[0021] In an exemplary embodiment, based on the foregoing solution, the obtaining module is specifically configured as follows: Calculate the ratio between the number of elements of each type of promotion element and the number of promotion elements included in the image to obtain a first initial saliency; calculate a second initial saliency according to the number of central promotion elements included in each type of promotion element; calculate the ratio between the total area of each type of promotion element in the image and the total area of the image to obtain a third initial saliency; calculate the ratio between the color difference value corresponding to each type of promotion element and the set maximum color value to obtain a fourth initial saliency.

[0022] In an exemplary embodiment, based on the foregoing solution, the pushing module is specifically configured as follows: If the pushing quality of the multimedia information is lower than the set pushing quality threshold, generate an optimization prompt message; send the optimization prompt message to the pushing request party corresponding to the multimedia information, so that the pushing request party optimizes the multimedia information according to the optimization prompt message; receive the optimized multimedia information sent by the pushing request party, and push the optimized multimedia information.

[0023] In an exemplary embodiment, based on the foregoing solution, the apparatus further includes a threshold determination module, configured to: obtain the information conversion rates corresponding to multiple pieces of historical multimedia information respectively; wherein the push time of the historical multimedia information is earlier than the push time of the multimedia information to be pushed; screen out target historical multimedia information from the multiple pieces of historical multimedia information whose corresponding information conversion rates are greater than a set conversion rate threshold; and determine the push quality threshold according to the push quality of the target historical multimedia information.

[0024] In an exemplary embodiment, based on the foregoing solution, the calculation module is specifically configured to: obtain the identification information of the promotion element and the theme description information of the multimedia information; if the identification information and the theme description information are texts, encode the identification information to obtain an identification information code, and encode the theme description information to obtain a theme description information code; calculate the distance between the identification information code and the theme description information code, and calculate the relevance between the promotion element and the multimedia information according to the distance; wherein the relevance is negatively correlated with the distance.

[0025] In an exemplary embodiment, based on the foregoing solution, the calculation module is specifically configured to: obtain at least one of the title, text summary, and category to which the push object belongs corresponding to the multimedia information, to obtain the theme description information of the multimedia information; and obtain the name of the object to which the promotion element belongs, to obtain the identification information of the promotion element.

[0026] In an exemplary embodiment, based on the foregoing solution, the acquisition module is specifically configured to: if the multimedia information is audio information, obtain the auditory salience feature information of the promotion element in the audio information; wherein the auditory salience feature information includes at least one of the duration, volume, position of the promotion element in the audio information, and the volume difference between the audio frame corresponding to the promotion element and the adjacent audio frames.

[0027] According to one aspect of the embodiments of the present application, there is provided an electronic device, including:

[0028] One or more processors;

[0029] A storage device for storing one or more computer programs, which when executed by the one or more processors, cause the electronic device to implement the method as described above.

[0030] According to one aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, which when executed by a processor of an electronic device, causes the electronic device to implement the method as described above.

[0031] According to one aspect of the embodiments of the present application, there is provided a computer program product including a computer program which, when executed by a processor, implements the method described above.

[0032] In the technical solution provided by the embodiments of the present application, first, the multimedia information to be pushed is obtained, and the multimedia information is detected to obtain the promotion elements included in the multimedia information. Then, the correlation between the promotion elements and the multimedia information is calculated, and the significant feature information of the promotion elements in the multimedia information is obtained. Based on the significant feature information, the significance degree of the promotion elements in the multimedia information is calculated. Thus, according to the correlation and the significance degree, the push quality of the multimedia information is calculated, and the multimedia information is pushed according to the push quality of the multimedia information. Compared with the related art, calculating the push quality of the multimedia information according to the correlation and the significance degree of the promotion elements included in the multimedia information and pushing the multimedia information according to the push quality can improve the accuracy of the push quality and the push effect of the multimedia information.

[0033] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of an implementation environment shown in an exemplary embodiment of the present application;

[0035] Figure 2 is a flowchart of a multimedia information processing method shown in an exemplary embodiment of the present application;

[0036] Figure 3A is a schematic diagram of an advertisement image shown in an exemplary embodiment of the present application;

[0037] Figure 3B is a schematic diagram of another advertisement image shown in an exemplary embodiment of the present application;

[0038] Figure 4 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0040] Figure 6 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0041] Figure 7 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0042] Figure 8 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0043] Figure 9 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0044] Figure 10 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0045] Figure 11 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0046] Figure 12 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0047] Figure 13 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0048] Figure 14 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0049] Figure 15 It is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application;

[0050] Figure 16 It is a block diagram of a multimedia information processing flow shown in an exemplary embodiment of the present application;

[0051] Figure 17 It is a schematic diagram of a multimedia information processing device shown in an exemplary embodiment of the present application;

[0052] Figure 18 It shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present application. Detailed implementation manners

[0053] Now, the exemplary embodiments will be described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as being limited to these examples; on the contrary, these embodiments are provided so that the present application is more comprehensive and complete, and the concept of the exemplary embodiments can be fully conveyed to those skilled in the art.

[0054] In addition, the features, structures, or characteristics described in this application can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to enable a full understanding of the embodiments of this application. However, those skilled in the art should be aware that when implementing the technical solutions of this application, not all the detailed features in the embodiments are required, one or more specific details can be omitted, or other methods, elements, devices, steps, etc. can be adopted.

[0055] In the embodiments of this application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of that module or unit.

[0056] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0057] The flowcharts shown in the drawings are only exemplary illustrations and do not necessarily include all the content and operations / steps, nor do they have to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0058] It should also be noted that: "a plurality of" mentioned in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0059] The following provides a detailed introduction to the technical solutions of the embodiments of this application:

[0060] In the related art, usually, only whether the multimedia information violates relevant regulations and whether the size of the multimedia information matches the size of the display position are considered to evaluate the multimedia information, resulting in a poor push effect of the multimedia information. Based on this, embodiments of the present application provide a multimedia information processing method, apparatus, electronic device, storage medium, and program product, which can detect the push quality of multimedia information, improve the accuracy of the push quality of multimedia information, and thus improve the push effect.

[0061] Please refer to Figure 1 , Figure 1 FIG. is a schematic diagram of an implementation environment related to the present application. The implementation environment includes a terminal device 110 and a server 120. The terminal device 110 and the server 120 communicate with each other through a wired or wireless network. The terminal device 110 can upload its own data to the server 120 or obtain data from the server 120.

[0062] Among them, the terminal device 110 may include, but is not limited to, a smart phone, a tablet, a laptop computer, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a remote driving terminal, etc.; the server 120 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The specific forms of the terminal device and the server are not limited herein.

[0063] It should be noted that Figure 1 the numbers of the terminal device 110 and the server 120 in

[0064] In an exemplary embodiment, the image processing method provided by the embodiments of the present application can be executed by the terminal device 110. Exemplarily, the terminal device 110 can first obtain the multimedia information to be pushed, detect the multimedia information to obtain the promotion elements included in the multimedia information, and then calculate the relevance between the promotion elements and the multimedia information, obtain the salient feature information of the promotion elements in the multimedia information, so as to calculate the salience of the promotion elements in the multimedia information according to the salient feature information, and thus calculate the push quality of the multimedia information according to the relevance and salience, and push the multimedia information according to the push quality of the multimedia information; wherein, determining the push quality of the multimedia information according to the relevance and salience of the promotion elements included in the multimedia information can improve the accuracy of the push quality, and pushing the multimedia information according to the push quality can improve the push effect of the multimedia information.

[0065] In another exemplary embodiment, the server 120 can have a function similar to that of the terminal device 110, so as to execute the image processing method provided by the embodiments of the present application. Exemplarily, the server 120 can first obtain the multimedia information to be pushed, detect the multimedia information to obtain the promotion elements included in the multimedia information, and then calculate the relevance between the promotion elements and the multimedia information, obtain the salient feature information of the promotion elements in the multimedia information, so as to calculate the salience of the promotion elements in the multimedia information according to the salient feature information, and thus calculate the push quality of the multimedia information according to the relevance and salience, and push the multimedia information according to the push quality of the multimedia information.

[0066] In another exemplary embodiment, the terminal device 110 and the server 120 can also jointly execute the image processing method provided by the embodiments of the present application. Exemplarily, after the terminal device 110 obtains the multimedia information to be pushed, it can send the multimedia information to the server 120. The server 120 first detects the multimedia information to obtain the promotion elements included in the multimedia information, and then calculates the relevance between the promotion elements and the multimedia information, obtains the salient feature information of the promotion elements in the multimedia information, so as to calculate the salience of the promotion elements in the multimedia information according to the salient feature information, and thus calculate the push quality of the multimedia information according to the relevance and salience, and push the multimedia information according to the push quality of the multimedia information.

[0067] The image processing method in the embodiments of this application can be applied to information push scenarios in various applications, such as applications like meetings, education, messaging, tourism, audiobooks, etc. These applications can be installed in in-vehicle terminals or other devices such as mobile phones. The applications include but are not limited to native applications, non-native applications, applets, web applications, etc. In the embodiments of this application, data related to users such as multimedia information is involved. When the method of this application is applied to specific products or technologies, it has obtained user permission or consent, and the extraction, use, and processing of relevant data comply with local safety standards and local laws and regulations.

[0068] See Figure 2 , Figure 2 is a flowchart of a multimedia information processing method shown in an exemplary embodiment of this application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in the implementation environment shown in Figure 1 the implementation environment shown, or can be executed by the server 120 in the implementation environment shown in Figure 1 the implementation environment shown, or, alternatively, can be jointly executed by the terminal device 110 and the server 120 in the implementation environment shown in Figure 1 the implementation environment shown.

[0069] As Figure 2 shown, in an exemplary embodiment, the multimedia information processing method may include steps S210 - step S240, which are introduced in detail as follows:

[0070] Step S210, obtain the multimedia information to be pushed, and detect the multimedia information to obtain the promotion elements included in the multimedia information.

[0071] It should be noted that the multimedia information to be pushed refers to any multimedia information that needs to be pushed to customers. Its information categories include but are not limited to text (for example, articles, soft articles, etc. that need to be promoted), images, videos, audio, etc. The form of the multimedia information to be pushed can be information for promoting products such as advertisements, posters, etc. The products it promotes include but are not limited to physical goods (such as cosmetics, daily necessities, clothes, etc.), virtual goods (such as game props, etc.), and works (such as novels, film and television dramas, music, etc.). The purpose or theme of the multimedia information to be pushed is usually to better promote the product to users, thereby attracting users to perform conversion behaviors such as consumption, use, and viewing. However, the layout of the promotion elements included in the multimedia information is different, resulting in different perceptions for users and different push effects or push qualities. For example, see Figure 3A and Figure 3B shown, Figure 3A and Figure 3B are both watch advertisements of a certain brand "R***D***", and Figure 3AAmong them, it includes promotional elements such as watches, lipsticks, and brand names. The watch is in the corner and has a small area, while the lipstick is in a more central position and has a larger area, which causes users to easily notice the lipstick and ignore the watch, reducing the push effect. And Figure 3B Among them, the watch is closer to the center of the image and has a larger area, so users can easily notice the watch.

[0072] Therefore, in order to detect the push quality of the multimedia information to be pushed, the multimedia information to be pushed can be detected to obtain the promotional elements included in the multimedia information. Among them, the promotional elements included in the multimedia information can be items, texts, etc. existing or described in the multimedia information. Items include, but are not limited to, animals, plants, daily necessities, etc. The multimedia information can be detected through natural language understanding technology, image understanding technology, etc. For example, speech recognition technology, semantic recognition technology, character recognition technology (e.g., OCR, i.e., Optical Character Recognition, optical character recognition), object detection technology, etc. Different types of multimedia information have different corresponding detection methods. For example, if the multimedia information is text, the text can be detected through semantic recognition technology; if the multimedia information is audio, the audio can be converted into text through speech recognition technology and then the text can be detected through semantic recognition technology; if the multimedia information is an image, the image can be detected through object detection technology, character recognition technology, etc.

[0073] Step S220, calculate the relevance between the promotional element and the multimedia information.

[0074] It should be understood that the theme of the multimedia information refers to the content that hopes to be conveyed to users and can represent the purpose of the multimedia information. The relevance between the promotional elements included in the multimedia information and the theme of the multimedia information is different, bringing different perceptions to users, thus affecting the push purpose and push effect. Therefore, the relevance between the promotional element and the multimedia information can be calculated. The more similar the promotional element is to the push object of the multimedia information, the higher the relevance between the promotional element and the multimedia information. For example, if the promotional element is a watch and the theme of the multimedia information is to recommend watches, then the relevance between the two is relatively high; if the promotional element is a lipstick and the theme of the multimedia information is to push mobile phones, then the relevance between the two is relatively low.

[0075] Among them, the specific method for calculating the relevance between the promotion element and the multimedia information can be flexibly set according to actual needs. In an optional example, the relevance between the promotion element and the multimedia information can be calculated based on the identification information of the promotion element and the theme description information of the multimedia information. The identification information of the promotion element is used to distinguish it from other promotion elements, and it can be the name of the promotion element, the promotion element label, the name of the item to which the promotion element belongs, etc. The promotion element label can be used to characterize the category of the promotion element, the name of the item to which the promotion element belongs, etc. The theme description information of the multimedia information refers to the information used to describe the theme of the multimedia information, including but not limited to the title of the multimedia information, the text summary, the category of the push object of the multimedia information, etc. The push object refers to the product being pushed. The title, text summary, and category of the push object can be set by the owner of the multimedia information (for example, if the multimedia information is an advertisement, the owner is the advertiser), or, to improve objectivity, the category of the push object can be determined from the multi-level categories included in the push system. For example, products can be first divided into furniture, electrical appliances, etc. Electrical appliances can be further divided into refrigerators, TVs, washing machines, etc. Refrigerators can be further divided into double-door refrigerators, French refrigerators, four-door refrigerators, etc. The identification information of the promotion element can be embodied in the form of text, etc., and the theme description information of the multimedia information can be embodied in the form of text, etc. The relevance can be characterized by scores, levels, etc. The more relevant the identification information of the promotion element is to the theme description information of the multimedia information, the higher the relevance between the promotion element and the multimedia information.

[0076] Step S230, obtain the salient feature information of the promotion element in the multimedia information, and calculate the salience of the promotion element in the multimedia information according to the salient feature information.

[0077] The saliency feature information of a promoted element refers to the feature information that can characterize whether the promoted element is salient in the multimedia information, and this information can affect the probability and degree to which a user perceives the promoted element. The saliency feature information includes, but is not limited to, auditory saliency feature information, visual saliency feature information, etc.; among them, the auditory saliency feature information targets the promoted element in the audio, and refers to the feature information that can characterize whether the promoted element is salient in auditory perception. This information can affect the probability and degree to which a user perceives the promoted element through hearing. The auditory saliency feature information includes at least one of the following feature information: the number of promoted elements, the duration, the volume, the order of appearance of the promoted element in the audio information, the volume difference between the audio frame corresponding to the promoted element and the adjacent audio frames of this audio frame, etc.; the visual saliency feature information targets the promoted element in the image, and refers to the feature information that can characterize whether the promoted element is salient in visual perception. This information can affect the probability and degree to which a user perceives the promoted element through vision. The visual saliency feature information includes at least one of the following attribute information: the number of promoted elements, the area, the color difference value between the promoted element and the background, the position of the promoted element (for example, whether the promoted element is located at the center of the image), etc.

[0078] The saliency of a promoted element in the multimedia information can characterize whether the promoted element is obvious in the multimedia information. Among them, the higher the saliency of the promoted element, the higher the probability and degree to which a user perceives the promoted element when viewing the multimedia information, and the lower the saliency, the lower the probability and degree to which a user perceives the promoted element when viewing the multimedia information. Therefore, the saliency of the promoted element in the multimedia information can be calculated based on the saliency feature information of the promoted element in the multimedia information, and the saliency can be characterized by scores, ranks, etc.

[0079] Among them, the stronger the auditory saliency or visual saliency corresponding to the saliency feature information of the promoted element, the higher the saliency of the promoted element. The specific method for calculating the saliency of the promoted element according to the saliency feature information of the promoted element can be flexibly set according to actual needs. For example, if the promoted element is a promoted element in the audio, the more the number of promoted elements, the longer the duration of the promoted element, the louder the volume of the promoted element, the more forward the position of the promoted element in the audio information, and the greater the volume difference between the audio frame corresponding to the promoted element and the adjacent audio frames, the higher the saliency of the promoted element; if the promoted element is a promoted element in the image, the more the number of promoted elements, the larger the area of the promoted element, the greater the color difference value between the promoted element and the background, and the closer the position of the promoted element to the center of the image, the higher the saliency of the promoted element.

[0080] Step S240, calculate the push quality of the multimedia information according to the relevance and the saliency, and push the multimedia information according to the push quality of the multimedia information.

[0081] The relevance between the promotion elements and the multimedia information, as well as the salience of the promotion elements, affect the push quality of the multimedia information. Among them, the higher the salience of the promotion elements with a higher relevance to the multimedia information, and the lower the salience of the promotion elements with a lower relevance to the multimedia information, the better the push quality of the multimedia information; the higher the salience of the promotion elements with a lower relevance to the multimedia information, and the lower the salience of the promotion elements with a higher relevance to the multimedia information, the lower the push quality of the multimedia information. Therefore, the push quality of the multimedia information can be calculated according to the relevance between the promotion elements and the multimedia information, as well as the salience of the promotion elements. Among them, the push quality can be characterized by a score or a level. The higher the score, the higher the quality, and the higher the level, the higher the quality.

[0082] After calculating the push quality, the multimedia information can be pushed according to the push quality. For example, if the push quality is high, the multimedia information can be directly pushed; if the push quality is low, the multimedia information can be optimized before being pushed.

[0083] In Figure 2 In the illustrated embodiment, compared with the related art, before pushing the multimedia information, determining the push quality of the multimedia information according to the relevance and salience of the promotion elements included in the multimedia information, and pushing the multimedia information according to the push quality can improve the accuracy of the push quality, thereby improving the push effect of the multimedia information.

[0084] In an exemplary embodiment, refer to Figure 4 , Figure 4 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the illustrated implementation environment, which can be executed by the terminal device 110 in the Figure 1 illustrated implementation environment, or can be executed by the server 120 in the Figure 1 illustrated implementation environment, or can also be jointly executed by the terminal device 110 and the server 120 in the Figure 1 illustrated implementation environment.

[0085] As Figure 4 shown, this method includes step S210, steps S410 - S440, and step S240, where the detailed introduction of steps S410 - S440 is as follows:

[0086] Step S410, if multiple categories of promotion elements are detected, calculate the relevance between each category of promotion element and the multimedia information according to the identification information corresponding to each category of promotion element and the theme description information of the multimedia information.

[0087] Under the condition that multiple promotion elements are detected from multimedia information, in order to improve the calculation efficiency, the multiple promotion elements can be classified to obtain multiple categories of promotion elements, and each category of promotion elements contains at least one promotion element; then, taking the promotion element category as a unit, the relevance is calculated, where the relevance between each category of promotion elements and the multimedia information can be calculated according to the relevance between the identification information corresponding to each category of promotion elements and the theme description information of the multimedia information.

[0088] Optionally, the promotion elements can be classified according to classification factors such as the name of the promotion element and the item to which the promotion element belongs.

[0089] Step S420, obtain the significant feature information of multiple feature categories of each category of promotion elements in the multimedia information.

[0090] Under the condition that multiple categories of promotion elements are detected, the significant feature information of each category of promotion elements in the multimedia information can be obtained. In order to improve the accuracy, multiple significant feature information of each category of promotion elements can be obtained. For example, if the promotion element is a promotion element in audio, the number of elements of each category of promotion elements, the total duration of each category of promotion elements, the number of promotion elements in the first half of the audio of each category of promotion elements, the average volume of each category of promotion elements, etc. can be obtained.

[0091] Step S430, calculate the initial salience corresponding to the significant feature information of each feature category.

[0092] The feature category refers to the category corresponding to the significant feature information. For each category of promotion elements, according to the significant feature information of each of its feature categories, the initial salience corresponding to the significant feature information of each feature category is calculated. For example, the initial salience corresponding to the number of elements of each category of promotion elements, the total duration of each category of promotion elements, the number of promotion elements in the first half of the audio of each category of promotion elements, and the average volume of each category of promotion elements can be calculated. Among them, the specific method of calculating the initial salience according to the significant feature information can be flexibly set according to actual needs. For example, the more the number of elements of each category of promotion elements, the higher the corresponding initial salience; the longer the total duration of each category of promotion elements, the higher the corresponding initial salience; the more the number of promotion elements in the first half of the audio of each category of promotion elements, the higher the corresponding initial salience; the higher the average volume of each category of promotion elements, the higher the corresponding initial salience.

[0093] Step S440, perform a summation operation on the initial saliences corresponding to the significant feature information of multiple feature categories to obtain the salience of each category of promotion elements in the multimedia information.

[0094] For each type of promotion element, add up the initial salience corresponding to the salient feature information of its multiple feature categories, and use the obtained sum as the salience corresponding to each type of promotion element.

[0095] It should be noted that Figure 4 The specific implementation details of the steps S210 and S240 shown can be referred to Figure 2 the steps S210 and S240 shown, and will not be elaborated here.

[0096] In Figure 4 the embodiment shown, taking the promotion element category as the unit, calculating the relevance and salience corresponding to each type of promotion element can reduce the calculation amount. Moreover, calculating the salience corresponding to each type of promotion element according to the salient feature information of multiple feature categories corresponding to each type of promotion element can improve the accuracy of the salience, thereby improving the accuracy of the push quality of the multimedia information.

[0097] In an exemplary embodiment, referring to Figure 5 , Figure 5 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in the Figure 1 implementation environment shown, or can be executed by the server 120 in the Figure 1 implementation environment shown, or, alternatively, can be jointly executed by the terminal device 110 and the server 120 in the Figure 1 implementation environment shown.

[0098] As Figure 5 shown, this method includes steps S210, S410 - S440, and steps S510 - S520. Among them, the detailed introduction of steps S510 - S520 is as follows:

[0099] Step S510, calculate the importance of each type of promotion element in the multimedia information according to the relevance between each type of promotion element and the multimedia information, and the salience of each type of promotion element in the multimedia information; wherein, the importance is positively correlated with the relevance corresponding to each type of promotion element and is positively correlated with the salience of each type of promotion element in the multimedia information.

[0100] Under the condition that multiple categories of promotion elements are detected, the importance of each category of promotion element in the multimedia information can be calculated according to the relevance and salience corresponding to each category of promotion element. Among them, the relevance corresponding to each category of promotion element is positively correlated with the importance, that is, the higher the relevance, the higher the importance; the salience corresponding to each category of promotion element is positively correlated with the importance, that is, the higher the salience, the higher the importance. Among them, the positive correlation relationship includes but is not limited to being proportional or other positive correlation relationships.

[0101] Optionally, before calculating the importance of each category of promotion element, the relevance corresponding to multiple categories of promotion elements can be normalized respectively, and the salience corresponding to multiple categories of promotion elements can be normalized respectively. Then, according to the normalized relevance and the normalized salience corresponding to each category of promotion element, the importance of each category of promotion element is calculated. Among them, the specific normalization method can be flexibly set according to actual needs. In an optional example, the salience can be normalized by the following formula:

[0102]

[0103] Among them, salienceScore′ i is the value representing the normalized salience of the i-th category of promotion element, salienceScore i is the value representing the salience of the i-th category of promotion element before normalization, salienceScore min is the minimum salience value among the saliences before normalization corresponding to multiple categories of promotion elements, salienceScore max is the maximum salience value among the saliences before normalization corresponding to multiple categories of promotion elements.

[0104] The relevance can be normalized by the following formula:

[0105]

[0106] Among them, cosinScore′ i is the value representing the normalized relevance of the i-th category of promotion element, cosinScore i is the value representing the relevance of the i-th category of promotion element before normalization, cosinScore min is the minimum relevance value among the relevances before normalization corresponding to multiple categories of promotion elements, cosinScore max is the maximum relevance value among the relevances before normalization corresponding to multiple categories of promotion elements.

[0107] Optionally, the product of the relevance and salience corresponding to each category of promotion elements can be used as the importance of each category of promotion elements.

[0108] In step S520, perform a summation operation on the importance corresponding to multiple categories of promotion elements respectively to obtain the push quality of the multimedia information.

[0109] Use the sum of the importance corresponding to multiple categories of promotion elements respectively as the push quality of the multimedia information.

[0110] In an optional example, if the product of the relevance and salience corresponding to each category of promotion elements is used as the importance of each category of promotion elements, the push quality of the multimedia information can be calculated by the following formula:

[0111]

[0112] Among them, Score is a numerical value representing the push quality of the multimedia information, and n is the number of categories of promotion elements detected from the multimedia information.

[0113] It should be noted that Figure 5 In the illustrated embodiment, taking the category of promotion elements as a unit, calculate the relevance, salience and importance corresponding to each category of promotion elements, and then perform a summation operation according to the importance corresponding to multiple categories of promotion elements respectively to obtain the push quality of the multimedia information; in other embodiments, under the condition of detecting multiple promotion elements, it is also possible to calculate the relevance and salience corresponding to each promotion element, and calculate the importance of each promotion element according to the relevance and salience corresponding to each promotion element, and then perform a summation operation on the importance corresponding to multiple promotion elements respectively to obtain the push quality of the multimedia information. Figure 5 The specific implementation details of step S210 shown can be referred to Figure 2 step S210 shown, Figure 5 The specific implementation details of steps S410 - S440 shown can be referred to Figure 4 steps S410 - S440 shown, which will not be elaborated here.

[0114] In Figure 5 the illustrated embodiment, first calculate the importance of each category of promotion elements according to the relevance between each category of promotion elements and the multimedia information, and the salience of each category of promotion elements, and then perform a summation operation on the importance corresponding to multiple categories of promotion elements respectively to obtain the push quality of the multimedia information, which can improve the accuracy of the push quality.

[0115] In an exemplary embodiment, refer to Figure 6 , Figure 6The flowchart of the multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be Figure 1 executed by the terminal device 110 in the implementation environment shown, or can be Figure 1 executed by the server 120 in the implementation environment shown, or, alternatively, can be Figure 1 jointly executed by the terminal device 110 and the server 120 in the implementation environment shown.

[0116] As Figure 6 shown, the method includes step S210, steps S410 - S430, steps S610 - S620, and step S240. Among them, the detailed introduction of steps S610 - S620 is as follows:

[0117] Step S610, look up the weights corresponding to various feature categories respectively for the category of the multimedia information from the set weight dataset.

[0118] For different categories of multimedia information, the influence of the significant feature information of the same feature category on the significance of the promotion element is different. For example, for an advertising image for promoting a panda doll, since the color of the panda doll is usually black and white, the color difference value has a relatively small impact on the significance of the panda doll, while the area has a relatively large impact on the significance of the panda doll; another example is that for an advertising image for promoting a refrigerator, since the area occupied by the refrigerator is usually large, the area has a relatively small impact on the significance of the refrigerator, while the color difference value has a relatively large impact on the significance of the parallel.

[0119] Therefore, a weight dataset can be preset. The weight dataset contains the weight data corresponding to different categories of multimedia information respectively, and the weight data corresponding to each category of multimedia information contains the weights corresponding to various feature categories respectively.

[0120] From the set weight dataset, the feature data corresponding to the category of the multimedia information to be pushed can be found. This feature data contains the weights corresponding to various feature categories respectively. Among them, the category of the multimedia information can be classified according to classification factors such as the push product and information category corresponding to the multimedia information.

[0121] Step S620, perform a summation operation on the initial saliencies corresponding to the significant feature information of various feature categories respectively according to the weights corresponding to various feature categories, to obtain the saliency of each category of promotion element in the multimedia information.

[0122] For each type of promotion element, according to the respective weights of multiple feature categories, perform a summation operation on the initial saliencies corresponding to the salient feature information of multiple feature categories, and use the obtained value as the saliency corresponding to each type of promotion element.

[0123] It should be noted that Figure 6 The specific implementation details of the steps S210 and S240 shown can be referred to Figure 2 the steps S210 and S240 shown Figure 6 The specific implementation details of the steps S410 - S430 shown can be referred to Figure 4 the steps S410 - S430 shown, which will not be elaborated here.

[0124] In Figure 6 the embodiment shown, determine the respective weights of multiple feature categories according to the category of the multimedia information, and thus perform a summation operation on the initial saliencies corresponding to the salient feature information of multiple feature categories according to the respective weights of multiple feature categories, to obtain the saliency corresponding to each type of promotion element, which can improve the accuracy of the saliency, and further improve the accuracy of the push quality of the multimedia information.

[0125] In an exemplary embodiment, refer to Figure 7 , Figure 7 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in Figure 1 the implementation environment shown, or can be executed by the server 120 in Figure 1 the implementation environment shown, or, can also be jointly executed by the terminal device 110 and the server 120 in Figure 1 the implementation environment shown.

[0126] As Figure 7 shown, this method includes steps S210, S410, steps S710 - S720, steps S430 - S440, and step S240, where the detailed introduction of steps S710 - S720 is as follows:

[0127] Step S710, if the multimedia information is an image, obtain the number of elements of each type of promotion element included in the image, and find the central promotion element located at the center of the image from each type of promotion element.

[0128] Under the condition that the multimedia information is an image, multiple types of promotion elements can be detected from the image.

[0129] The number of promotion elements, whether the promotion elements are located at the center of the image, the area of the promotion elements, and whether the color of the promotion elements is prominent will all affect the saliency of the promotion elements, thereby affecting the probability and degree to which users can view the promotion elements visually. Therefore, under the condition that the multimedia information contains an image, the number of promotion elements of each category contained in the image can be obtained, and the number of central promotion elements can be found from each category of promotion elements.

[0130] Among them, the central promotion element refers to the promotion element located at the center of the image, including but not limited to the promotion element containing the center point of the image, and the promotion element whose distance from the center point of the image is less than the set distance threshold. This distance threshold can be determined according to the length and width of the image. For example, it can be 10% of the width, etc.

[0131] Step S720: Calculate the total area of each category of promotion elements in the image, and obtain the color difference value between each category of promotion elements and the background of the image.

[0132] The area of the promotion element in the image can represent the size of the promotion element in the picture. This area can be the area enclosed by the edge of the promotion element or the area occupied by the rectangular detection frame of the promotion element; the total area of each category of promotion elements in the image refers to the sum of the areas of the promotion elements belonging to this category.

[0133] The color difference value between the promotion element and the background of the image refers to the color difference between the promotion element and the background, which can represent whether the color of the promotion element is prominent; among them, the specific calculation method of the color difference value between the promotion element and the background of the image can be flexibly set according to actual needs. For example, it can be calculated according to steps S820 - S840.

[0134] The color difference value between each category of promotion elements and the background of the image refers to the color difference between each category of promotion elements and the background, which can represent whether the color of this category of promotion elements is prominent. The specific calculation method can be flexibly set according to actual needs. In an optional example, if each category of promotion elements contains multiple promotion elements, then the area to which each promotion element belongs and the other areas except the area to which each promotion element belongs can be found from the rectangular detection frame of each promotion element. Calculate the average value of the color values of the pixel points contained in the areas to which multiple promotion elements belong, and calculate the average value of the color values of the pixel points contained in multiple other areas. Take the difference between the two average values as the color difference value corresponding to each category of promotion elements. Among them, the area to which the promotion element belongs refers to the area enclosed by the edge of the promotion element. The edge of the promotion element can be detected by edge detection technology. Among them, each pixel point corresponds to a color value, and the color mode corresponding to the color value includes but not limited to the RGB mode, the CMYK mode, etc.

[0135] It should be noted that Figure 7 in [example], taking the acquisition of the number of promotion elements, the central promotion element first, and then the total area and the color difference value as an example for illustration, in actual applications, the order of acquisition of these four types of data is not restricted; Figure 7 For the specific implementation details of the steps S210 and S240 shown, reference can be made to Figure 2 the steps S210 and S240 shown, Figure 7 For the specific implementation details of the steps S410, S430 - S430 shown, reference can be made to Figure 4 the steps S410, S430 - S430 shown, which will not be elaborated here.

[0136] In Figure 7 the embodiment shown, under the condition that the multimedia information is an image, taking the number of promotion elements of each category of promotion elements, the central promotion elements included in each category of promotion elements, the total area of each category of promotion elements, and the color difference value between each category of promotion elements and the background of the image as the saliency feature information can improve the accuracy of saliency, and further improve the accuracy of the push quality of the multimedia information.

[0137] In an exemplary embodiment, refer to Figure 8 , Figure 8 which is the flowchart of the multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in the Figure 1 implementation environment shown, or can be executed by the server 120 in the Figure 1 implementation environment shown, or, alternatively, can be jointly executed by the terminal device 110 and the server 120 in the Figure 1 implementation environment shown.

[0138] As Figure 8 shown, this method includes steps S210, S410, S710, S810 - S840, S430 - S440, and step S240, where the detailed introduction of steps S810 - S840 is as follows:

[0139] Step S810, calculate the total area of each category of promotion elements in the image.

[0140] Among them, the calculation method of the total area of each category of promotion elements can refer to the foregoing embodiment, which will not be elaborated here.

[0141] Step S820: If there are multiple promotion elements in each category of promotion elements, then for each promotion element included in the multiple promotion elements, find the area to which each promotion element belongs and the other areas except the area to which each promotion element belongs in the rectangular detection frame of each promotion element.

[0142] In the process of detecting the promotion elements included in the image, the rectangular detection frame corresponding to the promotion element can be obtained. The detection frame contains the promotion element. Under the condition that there are multiple promotion elements in each category of promotion elements, for each of these promotion elements, the area to which the promotion element belongs and the other areas in the rectangular detection frame before the area to which the promotion element belongs can be found from the rectangular detection frame of the promotion element.

[0143] Step S830: Calculate the average value of the color values of the multiple pixels included in the area to which each promotion element belongs to obtain the first color value, and calculate the average value of the color values of the multiple pixels included in the other areas to obtain the second color value.

[0144] For each promotion element, the first color value representing the color of the promotion element and the second color value representing the color of the background corresponding to the promotion element can be calculated. Among them, the average value of the color values of the multiple pixels included in the area to which the promotion element belongs can be used as the first color value, and the average value of the color values of the multiple pixels included in the other area corresponding to the promotion element can be used as the second color value.

[0145] In other embodiments, for each promotion element, the average value of the color values of the multiple pixels included in the rectangular detection frame corresponding to the promotion element can also be used as the first color value, and the average value of the color values of the multiple pixels included in the area outside the rectangular detection frame corresponding to the promotion element in the image can be used as the second color value.

[0146] Step S840: Calculate the difference between the first color value and the second color value to obtain the color difference value corresponding to each promotion element, and calculate the color difference value between each category of promotion elements and the background of the image according to the color difference values corresponding to the multiple promotion elements respectively.

[0147] For each promotion element, the difference between the corresponding first color value and the second color value is used as the color difference value corresponding to the promotion element, and the color difference values corresponding to the multiple promotion elements included in each category of promotion elements are used as the color difference value corresponding to each category of promotion elements. Among them, one of the maximum value, minimum value, median value, and average value can be selected from the color difference values corresponding to the multiple promotion elements respectively as the color difference value corresponding to each category of promotion elements.

[0148] It should be noted that Figure 8 The specific implementation details of the shown Step S210 and Step S240 can be referred toFigure 2 Steps S210 and S240 shown Figure 8 The specific implementation details of Steps S410 and S430 - S430 shown can be referred to Figure 4 Steps S410 and S430 - S430 shown Figure 8 The specific implementation details of Step S710 shown can be referred to Figure 7 Step S710 shown, which will not be elaborated here.

[0149] In Figure 8 In the embodiment shown, under the condition that each category of promotion elements contains multiple promotion elements, from the rectangular detection frames of each promotion element, find the area to which each promotion element belongs and the other areas except the area to which each promotion element belongs, calculate the average value of the color values of the multiple pixels included in the area to which each promotion element belongs to obtain the first color value, and calculate the average value of the color values of the multiple pixels included in the other areas to obtain the second color value. Take the difference between the first color value and the second color value as the color difference value corresponding to each promotion element, and calculate the color difference value between each category of promotion elements and the background of the image according to the color difference values corresponding to the multiple promotion elements respectively, which can reduce the data processing volume, improve the accuracy of the color difference value, and further improve the accuracy of the push quality of the multimedia information.

[0150] In an exemplary embodiment, refer to Figure 9 , Figure 9 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in the Figure 1 implementation environment shown, or can be executed by the server 120 in the Figure 1 implementation environment shown, or, alternatively, can be jointly executed by the terminal device 110 and the server 120 in the Figure 1 implementation environment shown.

[0151] As Figure 9 shown, this method includes Steps S210, S410, S710 - S720, S910 - S940, S440, and Step S240, where the detailed introduction of Steps S910 - S940 is as follows:

[0152] Step S910, calculate the ratio between the number of elements of each category of promotion elements and the number of promotion elements included in the image to obtain the first initial saliency.

[0153] Under the condition that the multimedia information is an image and the saliency feature information includes the number of elements of each type of promotion element, the ratio between the number of elements of each type of promotion element and the number of promotion elements included in the image can be used as the first initial saliency corresponding to each type of promotion element.

[0154] Step S920: Calculate a second initial saliency according to the number of central promotion elements included in each type of promotion element.

[0155] The second initial saliency corresponding to each type of promotion element is positively correlated with the number of central promotion elements included in each type of promotion element. That is to say, the more the number of central promotion elements corresponding to each type of promotion element, the higher the corresponding second initial saliency, that is, the more prominent. In an optional example, the number of central promotion elements included in each type of promotion element can be used as the second initial saliency.

[0156] Step S930: Calculate the ratio between the total area of each type of promotion element in the image and the total area of the image to obtain a third initial saliency.

[0157] The ratio between the total area of each type of promotion element in the image and the total area of the image is used as the third initial saliency corresponding to each type of promotion element.

[0158] Step S940: Calculate the ratio between the color difference value corresponding to each type of promotion element and the set maximum color value to obtain a fourth initial saliency.

[0159] The set maximum color value refers to a color value set in advance, which can be the maximum color value in the corresponding color mode. For example, for the RGB color mode, the maximum color value is 255.

[0160] The ratio between the color difference value corresponding to each type of promotion element and the set maximum color value is used as the fourth initial saliency corresponding to each type of promotion element.

[0161] After obtaining the first initial saliency, the second initial saliency, the third initial saliency, and the fourth initial saliency corresponding to each type of promotion element, a summation operation (or, weighted summation operation) can be performed on the first initial saliency, the second initial saliency, the third initial saliency, and the fourth initial saliency to obtain the saliency corresponding to each type of promotion element in the image. Then, according to the saliencies and relevance degrees corresponding to multiple types of promotion elements in the image, the push quality of the image can be calculated.

[0162] It should be noted that in this embodiment, no restriction is imposed on the execution order of steps S910 - S940; Figure 9 The specific implementation details of the shown steps S210 and S240 can be referred toFigure 2 Steps S210 and S240 shown Figure 9 The specific implementation details of steps S410 and S430 shown can be referred to Figure 4 Steps S410 and S430 shown, which will not be elaborated here.

[0163] In Figure 9 In the embodiment shown, taking the ratio between the number of elements of each type of promotion element and the number of promotion elements included in the image as the first initial saliency, calculating the second initial saliency according to the number of central promotion elements included in each type of promotion element, taking the ratio between the total area of each type of promotion element and the total area of the image as the third initial saliency, and taking the ratio between the color difference value corresponding to each type of promotion element and the set maximum color value as the fourth initial saliency can improve the accuracy and comparability of the saliency, and further improve the accuracy of the push quality of the multimedia information.

[0164] Refer to Figure 10 , Figure 10 is a flowchart of a multimedia information processing method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in the implementation environment shown, or can be executed by the server 120 in the implementation environment shown, or can also be jointly executed by the terminal device 110 and the server 120 in the implementation environment shown. Figure 1 the implementation environment shown, or can be executed by the server 120 in the implementation environment shown, or can also be jointly executed by the terminal device 110 and the server 120 in the implementation environment shown. Figure 1 the implementation environment shown, or can be jointly executed by the terminal device 110 and the server 120 in the implementation environment shown. Figure 1 the implementation environment shown.

[0165] As Figure 10 shown, in an exemplary embodiment, the multimedia information processing method may include steps S210 - S230, and steps S1010 - S1030, which are introduced in detail as follows:

[0166] Step S1010: Calculate the push quality of the multimedia information according to the relevance and saliency. If the push quality of the multimedia information is lower than the set push quality threshold, an optimization prompt message is generated.

[0167] After calculating the push quality of the multimedia information, the push quality can be compared with the set push quality threshold. If the push quality of the multimedia information is lower than the set push quality threshold, it indicates that the push effect of the multimedia information may be poor. Therefore, an optimization prompt message indicating to optimize the multimedia information can be generated.

[0168] Optionally, the optimization prompt information may further include optimization suggestions. For example, if the salience of a promotion element with a relatively high relevance to the multimedia information is low, the optimization prompt information may include an optimization suggestion to increase the salience of the promotion element.

[0169] Optionally, if the push quality of the multimedia information is higher than or equal to a set push quality threshold, the multimedia information may be pushed.

[0170] Step S1020: Send the optimization prompt information to the push requester corresponding to the multimedia information, so that the push requester optimizes the multimedia information according to the optimization prompt information.

[0171] The push requester refers to the party that requests the push of the multimedia information, which may be the owner of the multimedia information. For example, if the multimedia information is an advertisement, the push requester may be the advertiser, etc.

[0172] Send the optimization prompt information to the push requester, so that the push requester optimizes the multimedia information according to the optimization prompt information.

[0173] Step S1030: Receive the optimized multimedia information sent by the push requester and push the optimized multimedia information.

[0174] After receiving the optimized multimedia information sent by the push requester, the optimized multimedia information can be pushed.

[0175] Optionally, after receiving the optimized multimedia information sent by the push requester, the push quality of the optimized multimedia information may be calculated first. If the push quality is higher than or equal to the push quality threshold, the optimized multimedia information is pushed; if the push quality is lower than the push quality threshold, the optimization prompt information is continuously generated and optimized; or, if the push quality is lower than the push quality threshold, but the number of optimization times is greater than the set number threshold, or the push quality sent by the push requester is received, the optimized multimedia information is pushed.

[0176] It should be noted that Figure 10 The specific implementation details of the steps S210 - S230 shown can be referred to Figure 2 The steps S210 - S230 shown, which will not be elaborated here.

[0177] In Figure 10 In the embodiment shown, under the condition that the push quality of the multimedia information is lower than the set push quality threshold, the push requester can be instructed to optimize the multimedia information through the optimization prompt information, and the optimized multimedia information is pushed to improve the push effect.

[0178] In an exemplary embodiment, refer toFigure 11 , Figure 11 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown in Figure 1 and can be executed by the terminal device 110 in the implementation environment shown in Figure 1 , or can be executed by the server 120 in the implementation environment shown in Figure 1 , or can be jointly executed by the terminal device 110 and the server 120 in the implementation environment shown in

[0179] As Figure 11 shown, in an exemplary embodiment, the multimedia information processing method may include steps S1110 - S1130, steps S210 - S230, and steps S1010 - S1030. The detailed introduction of steps S1110 - S1130 is as follows:

[0180] Step S1110, obtain the information conversion rates corresponding to multiple historical multimedia information respectively; wherein, the push time of the historical multimedia information is earlier than the push time of the multimedia information to be pushed.

[0181] The information conversion rate of multimedia information refers to the ratio between the number of display times with conversion behavior and the total display times of the multimedia information, and is used to characterize how many times the conversion behavior has occurred among the display times of the multimedia information. The conversion behavior includes, but is not limited to, behaviors such as collection, like, forward, purchase, etc.

[0182] The push time of the historical multimedia information is earlier than the push time of the multimedia information to be pushed. To ensure the conversion effect of the multimedia information, the information conversion rates corresponding to multiple historical multimedia information can be obtained.

[0183] Among them, to improve the real - time performance, the information conversion rates corresponding to multiple historical multimedia information pushed within a historical time period adjacent to the current time can be obtained.

[0184] To improve the accuracy, multiple historical multimedia information with the same or similar category as the multimedia information to be pushed can be obtained.

[0185] Step S1120, screen out the target historical multimedia information whose corresponding information conversion rate is greater than the set conversion rate threshold from multiple historical multimedia information.

[0186] The set conversion rate threshold is used to evaluate the conversion effect of the multimedia information, and its specific value can be flexibly set according to actual needs. The target historical multimedia information refers to the historical multimedia information whose corresponding information conversion rate is greater than the set conversion rate threshold, that is, the multimedia information with a better conversion effect.

[0187] Step S1130: Determine a push quality threshold according to the push quality of the target historical multimedia information.

[0188] Determine a push quality threshold according to the push quality of the target historical multimedia information. Among them, the push quality threshold is positively correlated with the push quality of the target historical multimedia information. That is to say, the higher the push quality of the target historical multimedia information, the higher the push quality threshold.

[0189] In an optional example, if there is one target historical multimedia information, the push quality of this target historical multimedia information can be used as the push quality threshold; if there are multiple target historical multimedia information, the minimum value or the average value can be selected from the push qualities corresponding to the multiple target historical multimedia information as the push quality threshold.

[0190] It should be noted that Figure 11 The specific implementation details of the steps S210 - S230 shown can be referred to Figure 2 the steps S210 - S230 shown, Figure 11 The specific implementation details of the steps S1010 - S1030 shown can be referred to Figure 10 the steps S1010 - S1030 shown, which will not be elaborated here.

[0191] In Figure 11 the embodiment shown, determining the push quality threshold according to the push quality of the historical multimedia information with a higher conversion rate can improve the conversion rate of the multimedia information.

[0192] In an exemplary embodiment, refer to Figure 12 , Figure 12 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by the terminal device 110 in Figure 1 the implementation environment shown, or can be executed by the server 120 in Figure 1 the implementation environment shown, or, can also be jointly executed by the terminal device 110 and the server 120 in Figure 1 the implementation environment shown.

[0193] As Figure 12 shown, this method includes steps S210, steps S1210 - S1230, and steps S230 - S240. Among them, the detailed introduction of steps S1210 - S1230 is as follows:

[0194] Step S1210: Obtain the identification information of the promotion element and the theme description information of the multimedia information.

[0195] The relevance between the promotion element and the multimedia information can be calculated based on the identification information of the promotion element and the theme description information of the multimedia information. Therefore, the identification information of the promotion element and the theme description information of the multimedia information can be obtained first.

[0196] Step S1220: If the identification information and the multimedia information are text, encode the identification information to obtain the identification information encoding, and encode the theme description information to obtain the theme description information encoding.

[0197] Under the condition that both the identification information of the promotion element and the theme description information of the multimedia information are text, to improve the relevance calculation speed, the identification information and the theme description information can be encoded respectively to obtain the identification information encoding and the theme description information encoding. Among them, the encoding methods corresponding to the identification information and the theme description information are the same.

[0198] Step S1230: Calculate the distance between the identification information encoding and the theme description information encoding, and calculate the relevance between the promotion element and the multimedia information according to the distance; among them, the relevance is negatively correlated with the distance.

[0199] The distance between the identification information encoding and the theme description information encoding can represent the relevance between the promotion element and the multimedia information, and the relevance is negatively correlated with the distance. That is to say, the greater the distance, the smaller the relevance. The distance between the identification information encoding and the theme description information encoding can be the cosine distance. Optionally, the cosine similarity can be determined according to the cosine distance, and the cosine similarity can be used as the relevance between the promotion element and the multimedia information.

[0200] It should be noted that Figure 12 The specific implementation details of the steps S210, S230 - S240 shown can refer to Figure 2 the steps S210, S230 - S240 shown, which will not be elaborated here.

[0201] In Figure 12 the embodiment shown, under the condition that the identification information and the multimedia information are text, the identification information and the theme description information can be encoded respectively, and the relevance between the promotion element and the multimedia information can be determined according to the distance between the identification information encoding and the theme description information encoding, which can reduce the data processing volume and improve the accuracy of the relevance.

[0202] In an exemplary embodiment, refer to Figure 13 , Figure 13 is the flowchart of the multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be composed of Figure 1executed by the terminal device 110 in the illustrated implementation environment, or can also be Figure 1 executed by the server 120 in the illustrated implementation environment, or, can also be Figure 1 executed jointly by the terminal device 110 and the server 120 in the illustrated implementation environment.

[0203] As Figure 13 shown, in an exemplary embodiment, the multimedia information processing method may include step S210, steps S1310 - S1320, steps S1220 - S1230, and steps S230 - S240. Among them, the detailed introduction of steps S1310 - S1320 is as follows:

[0204] Step S1310, obtain at least one of the title, text summary, and category to which the push object belongs corresponding to the multimedia information, to obtain the theme description information of the multimedia information.

[0205] The title, text summary of the multimedia information, and the category to which the push object belongs can all represent the theme of the multimedia information. Therefore, the theme description information of the multimedia information may include at least one of the title, text summary, and category to which the push object belongs, etc.

[0206] Step S1320, obtain the name of the object to which the promotion element belongs, to obtain the identification information of the promotion element.

[0207] The object to which the promotion element belongs refers to items, animals, plants, etc. to which the promotion element belongs. Use the name of the object to which the promotion element belongs as the identification information of the promotion element.

[0208] It should be noted that Figure 13 For the specific implementation details of the shown steps S210, steps S230 - S240, reference can be made to Figure 2 the shown steps S210, steps S230 - S240, Figure 13 For the specific implementation details of the shown steps S1220 - S1230, reference can be made to Figure 12 the shown steps S1220 - S1230, which will not be elaborated here.

[0209] In Figure 13 the illustrated embodiment, use at least one of the title, text summary, and category to which the push object belongs corresponding to the multimedia information as the theme description information of the multimedia information, and use the name of the object to which the promotion element belongs as the identification information of the promotion element, thereby improving the accuracy of the relevance between the promotion element and the multimedia information.

[0210] In an exemplary embodiment, refer to Figure 14 , Figure 14The flowchart of the multimedia information processing method shown in another exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, which can be executed by Figure 1 the terminal device 110 in the implementation environment shown, or can be executed by Figure 1 the server 120 in the implementation environment shown, or, alternatively, can be jointly executed by Figure 1 the terminal device 110 and the server 120 in the implementation environment shown.

[0211] As Figure 14 shown, in an exemplary embodiment, the multimedia information processing method may include step S210 - step S220, step S1410 - step S1420, and step S240. Among them, the detailed introduction of step S1410 - step S1420 is as follows:

[0212] Step S1410, if the multimedia information is audio information, obtain the auditory saliency feature information of the promotion element in the audio information; wherein, the auditory saliency feature information includes at least one of the duration corresponding to the promotion element, the volume, the position of the promotion element in the audio information, and the volume difference between the audio frame corresponding to the promotion element and the adjacent audio frame.

[0213] Under the condition that the multimedia information is audio information, the auditory saliency feature information of the promotion element can be obtained; wherein, the auditory saliency feature information includes but is not limited to at least one of the duration corresponding to the promotion element, the volume, the position of the promotion element in the audio information, and the volume difference between the audio frame corresponding to the promotion element and the adjacent audio frame, etc.

[0214] Step S1420, calculate the saliency of the promotion element according to the saliency feature information.

[0215] After obtaining the auditory saliency feature information, the saliency of the promotion element can be calculated according to the auditory saliency feature information, so as to calculate the push quality of the audio information according to the saliency and relevance of the promotion element.

[0216] It should be noted that Figure 14 the specific implementation details of step S210 - step S230 and step S240 shown can be referred to Figure 2The steps S210 - S230 and S240 shown are not described herein again. In the foregoing embodiments, the specific calculation methods of the push quality corresponding to the audio information and the image information are specifically introduced. If the multimedia information is a video, the push quality of the audio information and the push quality of the image information included in the multimedia information can be calculated, and then the push quality of the audio information and the push quality of the image information are weighted and summed to obtain the push quality of the video information. Among them, in the process of calculating the push quality of the image information included in the video information, at least one target image can be extracted from multiple frames of images included in the video information, and the push quality corresponding to each of the at least one target image can be calculated. From the push quality corresponding to each of the at least one image, one of the maximum value, the minimum value, the average value, etc. is selected as the push quality of the image information included in the video information. Among them, the specific calculation method of the push quality corresponding to the target image is similar to the calculation method of the push quality of the multimedia information under the condition that the foregoing multimedia information is an image, and is not described herein again.

[0217] In Figure 14 the embodiment shown, when the multimedia information is audio information, the auditory saliency feature information includes at least one of the duration, volume, position of the promotion element in the audio information, and the volume difference between the audio frame corresponding to the promotion element and the adjacent audio frame. Calculating the saliency of the promotion element according to the auditory saliency feature information can improve the accuracy of the saliency.

[0218] In an exemplary embodiment, the multimedia information to be pushed is taken as an advertising image for illustration. Refer to Figure 15 , Figure 15 is a flowchart of a multimedia information processing method shown in another exemplary embodiment of the present application. As Figure 15 shown, the method includes steps S1501 - S1504, which are introduced in detail as follows:

[0219] Step S1501: Detect the advertising image to be pushed to obtain various types of promotion elements included in the advertising image and the visual saliency feature information of each type of promotion element.

[0220] In this embodiment, the multimedia information to be pushed is taken as an advertising image for illustration. In other embodiments, the multimedia information to be pushed can be an advertising video, etc.

[0221] After obtaining the advertisement image provided by the advertiser, the advertisement image to be pushed can be detected through object detection technology to detect the promotion elements contained in the advertisement image and the visual saliency feature information of the promotion elements. Among them, the object detection technology includes but is not limited to the Faster RCNN algorithm, etc. Generally speaking, it is necessary to detect "what is there", "how many", "where", "how big", "is the color conspicuous or not", etc. in the advertisement image.

[0222] Among them, "what is there" refers to which promotion elements are in the picture. The promotion elements in the image include two parts. One is which objects are in the picture, and the other is the theme of each text module in the picture. That is to say, in the process of detecting the promotion elements contained in the image, object recognition and text recognition need to be carried out. Among them, each promotion element can be identified by its corresponding promotion element label tagID. The promotion element label represents the category of the promotion element, which can be the name of the object to which the promotion element belongs, etc. The theme of the text module can be extracted through models such as NLP (Neuro-Linguistic Programming); "how many" refers to the number of promotion elements belonging to the same tagID; "where" refers to whether the position of the promotion element is at the center of the image. Among them, if the promotion element covers the center point of the image, it means that the promotion element is at the center of the image; "how big" refers to the area of the rectangular detection frame corresponding to the promotion element; "is the color conspicuous or not" refers to the difference between the color of the promotion element and the color of the image background, which can be represented by the color difference value between the promotion element and the background.

[0223] Step S1502, calculate the saliency of each category of promotion elements according to the visual saliency feature information of each category of promotion elements.

[0224] Optionally, the saliency of each category of promotion elements can be calculated through the following formula:

[0225]

[0226] Among them, salienceScore i refers to the saliency of the i-th category of promotion elements, Cnt i refers to the number of the i-th category of promotion elements, Cnt k refers to the number of the k-th category of promotion elements, n is the number of categories of promotion elements detected from the advertisement image, representing how many categories of promotion elements are detected from the advertisement image; area ij refers to the area of the rectangular detection frame corresponding to the j-th promotion element in the i-th category of promotion elements; imgArea refers to the area of the advertisement image; colerGap irefers to the color difference value between the promotion element of the i-th category and the image background; Center i refers to the number of promotion elements located at the center of the image among the promotion elements of the i-th category; α, β, γ, δ are weights, all of which are values greater than or equal to 0 and less than or equal to 1, and α + β + γ + δ = 1.

[0227] Step S1503: Calculate the relevance between each category of promotion element and the advertisement image according to the promotion element label of each category of promotion element and the theme description information of the advertisement image.

[0228] Among them, the promotion element label and the theme description information can be encoded respectively using the same encoding algorithm to obtain the promotion element label encoding and the theme description information encoding, and the relevance between each category of promotion element and the advertisement image is calculated according to the cosine distance between the promotion element label encoding and the theme description information encoding.

[0229] The theme description information of the advertisement image can be used to describe the advertised product.

[0230] Step S1504: Calculate the push quality of the advertisement image according to the relevance and salience corresponding to multiple categories of promotion elements.

[0231] Among them, it can be calculated according to the following formula

[0232]

[0233] That is to say, as shown in Figure 16 After obtaining the advertisement image, the image can be detected. During the detection process, promotion element recognition needs to be performed, including text recognition and object recognition; the coordinates of the rectangular detection frame of the promotion element need to be obtained to determine the area of the promotion element and whether the promotion element is located at the center of the image according to the coordinates; the colors of the promotion element and the background need to be obtained to calculate the color difference value; then, the relevance between the promotion element and the advertised product is calculated according to the detected data, the salience of the promotion element is calculated, and the push quality of the advertisement image is calculated through the relevance and salience.

[0234] It should be noted that Figure 15 The detailed processes involved in Step S1501 - Step S1507 shown in are described in the foregoing respective embodiments, so they will not be elaborated here.

[0235] In Figure 15 In the embodiment shown, the placement quality of the advertisement image can be accurately predicted, so as to guide the advertiser to improve the placement effect of the advertisement image according to a better advertisement image.

[0236] As shown in Figure 17 ,Figure 17 It is a block diagram of a multimedia information processing device shown in an exemplary embodiment of the present application. As Figure 17 shown, the device includes:

[0237] A detection module 1701, configured to obtain multimedia information to be pushed, detect the multimedia information, and obtain promotion elements included in the multimedia information;

[0238] A calculation module 1702, configured to calculate the relevance between the promotion elements and the multimedia information;

[0239] An acquisition module 1703, configured to obtain the saliency feature information of the promotion elements in the multimedia information, and calculate the saliency degree of the promotion elements in the multimedia information according to the saliency feature information;

[0240] A push module 1704, configured to calculate the push quality of the multimedia information according to the relevance and the saliency degree, and push the multimedia information according to the push quality of the multimedia information.

[0241] In an exemplary embodiment, based on the foregoing solution, the calculation module 1702 is specifically configured to: if multiple categories of promotion elements are detected, calculate the relevance between each category of promotion elements and the multimedia information according to the identification information corresponding to each category of promotion elements and the theme description information of the multimedia information;

[0242] The acquisition module 1703 is specifically configured to: obtain the saliency feature information of multiple feature categories of each category of promotion elements in the multimedia information; calculate the initial saliency degree corresponding to the saliency feature information of each feature category; perform a summation operation on the initial saliency degrees corresponding to the saliency feature information of multiple feature categories to obtain the saliency degree of each category of promotion elements in the multimedia information.

[0243] In an exemplary embodiment, based on the foregoing solution, the processing module is specifically configured to: calculate the importance degree of each category of promotion elements in the multimedia information according to the relevance between each category of promotion elements and the multimedia information, and the saliency degree of each category of promotion elements in the multimedia information; wherein, the importance degree is positively correlated with the relevance corresponding to each category of promotion elements, and is positively correlated with the saliency degree of each category of promotion elements in the multimedia information; perform a summation operation on the importance degrees corresponding to multiple categories of promotion elements to obtain the push quality of the multimedia information.

[0244] In an exemplary embodiment, based on the foregoing solution, the obtaining module 1703 is specifically configured to: search for the weights corresponding to various feature categories of the category of the multimedia information from a set weight dataset; perform a summation operation on the initial saliencies corresponding to the salient feature information of various feature categories according to the weights corresponding to various feature categories respectively, so as to obtain the saliency of each category of promotion elements in the multimedia information.

[0245] In an exemplary embodiment, based on the foregoing solution, the obtaining module 1703 is specifically configured to: if the multimedia information is an image, obtain the number of elements of each category of promotion elements included in the image, and find the central promotion element located at the center of the image from each category of promotion elements; calculate the total area of each category of promotion elements in the image, and obtain the color difference value between each category of promotion elements and the background of the image.

[0246] In an exemplary embodiment, based on the foregoing solution, the obtaining module 1703 is specifically configured to: if each category of promotion elements includes multiple promotion elements, find the area to which each promotion element belongs and other areas except the area to which each promotion element belongs from the rectangular detection frames of each promotion element included in the multiple promotion elements; calculate the average value of the color values of multiple pixels included in the area to which each promotion element belongs to obtain a first color value, and calculate the average value of the color values of multiple pixels included in other areas to obtain a second color value; calculate the difference between the first color value and the second color value to obtain the color difference value corresponding to each promotion element, and calculate the color difference value between each category of promotion elements and the background of the image according to the color difference values corresponding to multiple promotion elements respectively.

[0247] In an exemplary embodiment, based on the foregoing solution, the obtaining module 1703 is specifically configured to: calculate the ratio between the number of elements of each category of promotion elements and the number of promotion elements included in the image to obtain a first initial saliency; calculate a second initial saliency according to the number of central promotion elements included in each category of promotion elements; calculate the ratio between the total area of each category of promotion elements in the image and the total area of the image to obtain a third initial saliency; calculate the ratio between the color difference value corresponding to each category of promotion elements and a set maximum color value to obtain a fourth initial saliency.

[0248] In an exemplary embodiment, based on the foregoing solution, the pushing module 1704 is specifically configured to: if the pushing quality of the multimedia information is lower than a set pushing quality threshold, generate an optimization prompt message; send the optimization prompt message to the pushing request party corresponding to the multimedia information, so that the pushing request party optimizes the multimedia information according to the optimization prompt message; receive the optimized multimedia information sent by the pushing request party, and push the optimized multimedia information.

[0249] In an exemplary embodiment, based on the foregoing solution, the apparatus further includes a threshold determination module configured to: obtain the information conversion rates corresponding to multiple pieces of historical multimedia information respectively, where the push time of the historical multimedia information is earlier than the push time of the multimedia information to be pushed; screen out target historical multimedia information from the multiple pieces of historical multimedia information, where the information conversion rate corresponding to the target historical multimedia information is greater than a set conversion rate threshold; and determine a push quality threshold according to the push quality of the target historical multimedia information.

[0250] In an exemplary embodiment, based on the foregoing solution, the calculation module 1702 is specifically configured to: obtain the identification information of the promotion element and the theme description information of the multimedia information; if the identification information and the theme description information are text, encode the identification information to obtain an identification information code, and encode the theme description information to obtain a theme description information code; calculate the distance between the identification information code and the theme description information code, and calculate the relevance between the promotion element and the multimedia information according to the distance; where the relevance is negatively correlated with the distance.

[0251] In an exemplary embodiment, based on the foregoing solution, the calculation module 1702 is specifically configured to: obtain at least one of the title, text summary, and category to which the push object belongs corresponding to the multimedia information to obtain the theme description information of the multimedia information; and obtain the name of the object to which the promotion element belongs to obtain the identification information of the promotion element.

[0252] In an exemplary embodiment, based on the foregoing solution, the acquisition module 1703 is specifically configured to: if the multimedia information is audio information, obtain the auditory salience feature information of the promotion element in the audio information; where the auditory salience feature information includes at least one of the duration, volume, position of the promotion element in the audio information, and the volume difference between the audio frame corresponding to the promotion element and the adjacent audio frame.

[0253] It should be noted that Figure 17 The provided multimedia information processing apparatus and the multimedia information processing method provided in the above embodiments belong to the same concept. The specific manners in which each module and unit perform operations have been described in detail in the method embodiments and will not be repeated here.

[0254] An embodiment of the present application further provides an electronic device, including: one or more processors; a storage device for storing one or more computer programs, and when the one or more computer programs are executed by the one or more processors, the electronic device implements the multimedia information processing method provided in each of the above embodiments.

[0255] Figure 18 The structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown.

[0256] It should be noted that Figure 18 The computer system 1800 of the illustrated electronic device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0257] As Figure 18 shown, the computer system 1800 includes a central processing unit (CPU) 1801, which can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 1802 or the computer program loaded from the storage section 1808 into the random access memory (RAM) 1803, such as executing the multimedia information processing method in the above embodiments. In the RAM 1803, various computer programs and data required for system operation are also stored. The CPU 1801, ROM 1802, and RAM 1803 are connected to each other via a bus 1804. The input / output (I / O) interface 1803 is also connected to the bus 1804.

[0258] In some embodiments, the following components are connected to the I / O interface 1803: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1803 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed so that the computer program read from it can be installed into the storage section 1808 as needed.

[0259] Specifically, according to the embodiments of the present application, the computer program for implementing the multimedia information processing method can be carried on a computer-readable medium, and this computer program can be downloaded and installed from the network through the communication section 1809, and / or installed from the removable medium 1811.

[0260] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a computer program, and the computer program can be used by or combined with an instruction execution system, apparatus, or device. A computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer program contained in the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0261] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and a computer program.

[0262] The units involved in the embodiments of the present application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation on the units themselves in some cases.

[0263] On the other hand, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device implements the multimedia information processing method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist alone without being assembled into the electronic device.

[0264] On the other hand, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the multimedia information processing methods provided in the above various embodiments. Among them, the computer program can be stored in a computer-readable storage medium. The computer program product can be a computer program as a product, for example, an APP (Application, mobile phone application software), a web page, a small program, etc.; or, the computer program product can also be a storage medium, a device, a terminal, a virtual machine, etc. containing the computer program.

[0265] The above content is only a preferred exemplary embodiment of the present application and is not used to limit the implementation of the present application. Those of ordinary skill in the art can easily make corresponding changes or modifications according to the main idea and spirit of the present application. Therefore, the protection scope of the present application should be subject to the protection scope required by the claims.

Claims

1. A multimedia information processing method, characterized in that, The method includes: Obtain multimedia information to be pushed, and detect the multimedia information to obtain promotion elements included in the multimedia information; Calculate the relevance between the promotion elements and the multimedia information; Obtain the salient feature information of the promotion elements in the multimedia information, and calculate the salience of the promotion elements in the multimedia information according to the salient feature information; Calculate the push quality of the multimedia information according to the relevance and the salience, and push the multimedia information according to the push quality of the multimedia information.

2. The method according to claim 1, characterized in that The calculating the relevance between the promotion elements and the multimedia information includes: If multiple types of promotion elements are detected, calculate the relevance between each type of promotion element and the multimedia information according to the identification information corresponding to each type of promotion element and the theme description information of the multimedia information; The obtaining the salient feature information of the promotion elements in the multimedia information and calculating the salience of the promotion elements in the multimedia information according to the salient feature information includes: Obtain the salient feature information of multiple feature categories of each type of promotion element in the multimedia information; Calculate the initial salience corresponding to the salient feature information of each feature category; Perform a summation operation on the initial saliences corresponding to the salient feature information of the multiple feature categories to obtain the salience of each type of promotion element in the multimedia information.

3. The method according to claim 2, wherein The calculating the push quality of the multimedia information according to the relevance and the salience includes: Calculate the importance of each type of promotion element in the multimedia information according to the relevance between each type of promotion element and the multimedia information and the salience of each type of promotion element in the multimedia information; wherein, the importance is positively correlated with the relevance corresponding to each type of promotion element and is positively correlated with the salience of each type of promotion element in the multimedia information; Perform a summation operation on the importances corresponding to the multiple types of promotion elements to obtain the push quality of the multimedia information.

4. The method according to claim 2, wherein The performing a summation operation on the initial saliences corresponding to the salient feature information of the multiple feature categories to obtain the salience of each type of promotion element in the multimedia information includes: Search in the set weight dataset for the weights of the multiple feature categories corresponding to the category of the multimedia information; According to the weights of the multiple feature categories respectively, perform a summation operation on the initial saliences corresponding to the salient feature information of the multiple feature categories to obtain the salience of each type of promotion element in the multimedia information.

5. The method according to claim 2, wherein The obtaining the salient feature information of multiple feature categories of each type of promotion element in the multimedia information includes: If the multimedia information is an image, obtain the number of elements of each type of promotion element included in the image, and find the central promotion element located at the center of the image from each type of promotion element; Calculate the total area of each type of promotion element in the image, and obtain the color difference value between each type of promotion element and the background of the image.

6. The method according to claim 5, characterized in that, The obtaining the color difference value between each type of promotion element and the background of the image includes: If there are multiple promotion elements in each type of promotion element, find the area to which each promotion element belongs and other areas except the area to which each promotion element belongs from the rectangular detection frames of each promotion element included in the multiple promotion elements; Calculate the average value of the color values of the multiple pixels included in the area to which each promotion element belongs to obtain a first color value, and calculate the average value of the color values of the multiple pixels included in the other area to obtain a second color value; Calculate the difference between the first color value and the second color value to obtain the color difference value corresponding to each promotion element, and calculate the color difference value between each type of promotion element and the background of the image according to the color difference values corresponding to the multiple promotion elements respectively.

7. The method according to claim 5, wherein The calculating the initial saliency corresponding to the saliency feature information of each feature category includes: Calculate the ratio between the number of elements of each type of promotion element and the number of promotion elements included in the image to obtain a first initial saliency; Calculate a second initial saliency according to the number of central promotion elements included in each type of promotion element; Calculate the ratio between the total area of each type of promotion element in the image and the total area of the image to obtain a third initial saliency; Calculate the ratio between the color difference value corresponding to each type of promotion element and the set maximum color value to obtain a fourth initial saliency.

8. The method according to claim 1, wherein The pushing the multimedia information according to the pushing quality of the multimedia information includes: If the pushing quality of the multimedia information is lower than the set pushing quality threshold, generate an optimization prompt message; Send the optimization prompt message to the pushing request party corresponding to the multimedia information, so that the pushing request party optimizes the multimedia information according to the optimization prompt message; Receive the optimized multimedia information sent by the pushing request party and push the optimized multimedia information.

9. The method according to claim 8, characterized in that, The method further includes: Obtain the information conversion rates corresponding to multiple historical multimedia information respectively; wherein, the pushing time of the historical multimedia information is earlier than the pushing time of the multimedia information to be pushed; Screen out the target historical multimedia information from the multiple historical multimedia information whose corresponding information conversion rate is greater than the set conversion rate threshold; Determine the pushing quality threshold according to the pushing quality of the target historical multimedia information.

10. The method according to claim 1, characterized in that, The calculating the relevance between the promotion element and the multimedia information includes: Obtain the identification information of the promotion element and the theme description information of the multimedia information; If the identification information and the theme description information are texts, encode the identification information to obtain an identification information code, and encode the theme description information to obtain a theme description information code; Calculate the distance between the encoded identification information and the encoded subject description information, and calculate the relevance between the promotion element and the multimedia information according to the distance; wherein, the relevance is negatively correlated with the distance.

11. The method according to claim 10, characterized in that The obtaining the identification information of the promotion element and the subject description information of the multimedia information includes: Obtaining at least one of the title corresponding to the multimedia information, the text summary, and the category to which the push object belongs, to obtain the subject description information of the multimedia information; Obtaining the name of the object to which the promotion element belongs to obtain the identification information of the promotion element.

12. The method according to claim 1, characterized in that, The obtaining the salient feature information of the promotion element in the multimedia information includes: If the multimedia information is audio information, obtaining the auditory salient feature information of the promotion element in the audio information; wherein, the auditory salient feature information includes at least one of the duration corresponding to the promotion element, the volume, the position of the promotion element in the audio information, and the volume difference between the audio frame corresponding to the promotion element and the adjacent audio frames.

13. A multimedia information processing apparatus, characterized in that, The apparatus includes: A detection module configured to obtain the multimedia information to be pushed and detect the multimedia information to obtain the promotion element included in the multimedia information; A calculation module configured to calculate the relevance between the promotion element and the multimedia information; An acquisition module configured to obtain the salient feature information of the promotion element in the multimedia information and calculate the salience of the promotion element in the multimedia information according to the salient feature information; A push module configured to calculate the push quality of the multimedia information according to the relevance and the salience, and push the multimedia information according to the push quality of the multimedia information.

14. An electronic device, characterized in that, Comprising: One or more processors; A storage device for storing one or more computer programs, which when executed by the one or more processors, cause the electronic device to implement the method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, Having a computer program stored thereon, which when executed by a processor of an electronic device, causes the electronic device to implement the method according to any one of claims 1-12.

16. A computer program product, characterized in that, Including a computer program, which when executed by a processor, implements the method according to any one of claims 1-12.

Images