Information recommendation method and device, electronic equipment, storage medium and program product

Abstract

The application provides an image-based information recommendation method and device, electronic equipment, computer readable storage medium and computer program product; the method comprises: performing blank area identification processing on an information browsing interface comprising at least one information to obtain a plurality of blank areas in the information browsing interface; performing material identification processing on an image used for information recommendation to obtain a plurality of recommendation materials in the image; obtaining a priority of each recommendation material; selecting the recommendation materials in order from high to low priority; based on the selected recommendation materials, forming a material set corresponding to the blank area, wherein the size of at least one recommendation material included in the material set is adapted to the size of the blank area; and displaying the material set corresponding to the blank area in at least one blank area of the information browsing interface. Through the application, the recommendation flexibility and efficiency of image-based information recommendation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an image-based information recommendation method and device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Artificial intelligence (AI) is a theory, method and technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0003] In the related art, a recommendation information is displayed by using a pre-configured information bit, and the size of the recommendation information and the size of the information bit are greatly limited. There are many blank areas in an information browsing interface in the related art. However, the sizes of the blank areas are not suitable for being used as information bits to completely display the recommendation information, so that the display resources in the interface are wasted and the display scenarios of the recommendation information are limited. SUMMARY

[0004] The embodiments of the present application provide an image-based information recommendation method and device, an electronic device, a computer readable storage medium and a computer program product, which can improve the recommendation flexibility and efficiency of information recommendation based on images.

[0005] The technical solutions of the embodiments of the present application are implemented as follows:

[0006] The embodiments of the present application provide an image-based information recommendation method, which comprises the following steps:

[0007] Performing blank area identification processing on an information browsing interface including at least one information to obtain a plurality of blank areas in the information browsing interface;

[0008] Performing material identification processing on an image used for information recommendation to obtain a plurality of recommendation materials in the image;

[0009] Obtaining a priority of each recommendation material;

[0010] Selecting the recommendation materials in a descending order of the priorities, and composing a material set corresponding to the blank areas based on the selected recommendation materials, wherein the sum of the sizes of at least one recommendation material included in the material set is adapted to the size of the blank area;

[0011] Displaying the material set corresponding to the blank area in at least one blank area of the information browsing interface.

[0012] The embodiment of the present application provides a kind of information recommendation device based on image, comprising:

[0013] First regional module, for the information browsing interface including at least one information is carried out blank area identification processing, obtain multiple blank areas in the information browsing interface;

[0014] First material module, for the image for information recommendation is carried out material identification processing, obtain multiple recommended materials in the image;

[0015] First priority module, for obtaining the priority of each recommended material;

[0016] First composition module, for selecting the recommended material according to the priority from high to low order, based on the recommended material selected and the blank area corresponding material set is formed, wherein the size of at least one recommended material included in the material set The sum of the size of the blank area is adapted;

[0017] First display module, for displaying the material set corresponding to the blank area in at least one blank area of the information browsing interface.

[0018] In the above scheme, the first material module is also used for: the image is preprocessed to obtain a preprocessed image, wherein the preprocessing includes at least one of the following: image denoising processing, image smoothing processing, image transformation processing;The image features of the preprocessed image are extracted to obtain the image features of the preprocessed image;The image features are mapped to obtain the probability of each sample pixel block in the image corresponding to multiple material types;For each sample pixel block, the following processing is performed: the material type with the highest probability is determined as the material type corresponding to the sample pixel block;Adjacent sample pixel blocks corresponding to the same material type are merged to obtain recommended materials corresponding to the same material type.

[0019] In the above scheme, the first material module is also used for: the image features of the preprocessed image are extracted to obtain the three-dimensional features of each pixel block in the preprocessed image;Based on the spatial features in the three-dimensional features of each pixel block, the spatial sampling processing is carried out on multiple pixel blocks in the preprocessed image to obtain multiple sample pixel blocks;The three-dimensional features of the multiple sample pixel blocks are spliced into the image features.

[0020] In the above scheme, the first priority module is further configured to perform any one of the following processing: obtaining a priority configured in advance for each material type, and determining the priority of the material type of the recommended material as the priority of the recommended material; obtaining a feature parameter of the recommended material, and determining a priority positively correlated with the feature parameter, wherein the feature parameter comprises at least one of the following: an area of the recommended material, and a degree of interest of the object to be recommended for the recommended material.

[0021] In the above scheme, the first component module is further configured to: obtain at least one blank area meeting a blank area condition from the plurality of blank areas as a blank area to be filled; and perform the following processing for each blank area to be filled: querying a mapping table for a number of materials corresponding to a size of the blank area, sorting the plurality of recommended materials in descending order of the priority, and obtaining at least one recommended material meeting the number of materials in the order of sorting; and performing scaling processing on the at least one recommended material based on the size of the blank area, and grouping the at least one recommended material after the scaling processing into the set of materials.

[0022] In the above scheme, the first component module is further configured to: obtain at least one blank area meeting a blank area condition from the plurality of blank areas as a blank area to be filled; and perform the following processing for each blank area to be filled: sorting the plurality of recommended materials in descending order of the priority, and performing the following processing for each recommended material according to the result of sorting: when a fillable size of the blank area to be filled is not less than a size of the recommended material, taking the recommended material as a target material for filling the blank area to be filled; and when the fillable size of the blank area to be filled is less than the size of the recommended material, generating a set of materials corresponding to the blank area based on the target material that has been selected.

[0023] In the above scheme, the blank area condition comprises at least one of the following: the size of the blank area is greater than a size threshold; a distance between the position of the blank area and the center of the information browsing interface is not less than a first distance threshold; and a minimum distance between the blank area and a set of displayed materials is greater than a second distance threshold.

[0024] In the above scheme, the first material module is further configured to perform any one of the following processing for a plurality of candidate images: determining a degree of association of the candidate image with information content displayed in the information browsing interface, and taking a candidate image with the highest degree of association as the image for information recommendation; and determining a degree of interest of the object to be recommended for the candidate image, and taking a candidate image with the highest degree of interest as the image.

[0025] In the above scheme, the first area module is further configured to: acquire a preview interface image of the information browsing interface, the preview interface image being an interface image in which information content is displayed in the information browsing interface; identify pixels in the preview interface image that do not belong to the information content as background pixels; perform connected processing on the background pixels that are adjacent to each other of the plurality of background pixels to obtain a plurality of connected regions, and perform cropping processing on each of the connected regions to obtain a plurality of blank areas conforming to a preset shape.

[0026] In the above scheme, the first area module is further configured to: acquire a standard binarization result corresponding to the blank area; perform binarization processing on each pixel in the preview interface image to obtain a binarization result of each pixel; perform compensation processing on the binarization result of each pixel to obtain a compensated binarization result of each pixel; and identify a pixel corresponding to the compensated binarization result as the background pixel when the compensated binarization result is the same as the standard binarization result.

[0027] In the above scheme, the first area module is further configured to: determine an information color value of a representative pixel belonging to the information content and a blank color value of a representative pixel not belonging to the information content; when the blank color value is less than the information color value, take a zero value as the standard binarization result of the blank area; and when the blank color value is not less than the information color value, take a non-zero value as the standard binarization result of the blank area.

[0028] Embodiments of the present application provide an image-based information recommendation method, comprising:

[0029] Performing blank area identification processing on an information browsing interface in a terminal to obtain a plurality of blank areas in the information browsing interface;

[0030] Performing material identification processing on an image used for information recommendation to obtain a plurality of recommendation materials in the image;

[0031] Acquiring a priority of each of the recommendation materials;

[0032] Selecting the recommendation materials in order from high to low of the priority, and composing a material set from the selected recommendation materials, wherein a sum of sizes of at least one of the recommendation materials included in the material set is adapted to a size of the blank area;

[0033] Sending the material set corresponding to at least one of the blank areas to the terminal.

[0034] Embodiments of the present application provide an image-based information recommendation device, comprising:

[0035] The second region module is configured to perform blank region identification processing on an information browsing interface in the terminal to obtain a plurality of blank regions in the information browsing interface.

[0036] The second material module is configured to perform material identification processing on an image for information recommendation to obtain a plurality of recommended materials in the image.

[0037] The second priority module is configured to obtain a priority of each recommended material.

[0038] The second composition module is configured to select the recommended materials in a descending order of the priorities, and compose the blank region based on the selected recommended materials, wherein a sum of sizes of at least one recommended material included in the material set is adapted to a size of the blank region.

[0039] The sending module is configured to send the material set corresponding to at least one blank region to the terminal.

[0040] Embodiments of the present application provide a method for recommending information based on an image, comprising:

[0041] Displaying at least one information in an information browsing interface.

[0042] Displaying a material set in at least one blank region remaining after displaying the at least one information.

[0043] The material set includes at least one recommended material, and is obtained by identifying the image, a sum of sizes of at least one recommended material included in the material set is adapted to a size of the blank region, and the recommended materials are sequentially selected in a descending order of priorities of a plurality of recommended materials in the image.

[0044] Embodiments of the present application provide a device for recommending information based on an image, comprising:

[0045] The second display module is configured to display at least one information in an information browsing interface.

[0046] The second display module is further configured to display a material set in at least one blank region remaining after displaying the at least one information, wherein the material set includes at least one recommended material, and is obtained by identifying the image, a sum of sizes of at least one recommended material included in the material set is adapted to a size of the blank region, and the recommended materials are sequentially selected in a descending order of priorities of a plurality of recommended materials in the image.

[0047] Embodiments of the present application provide a method for recommending information based on an image, comprising:

[0048] displaying at least part of content of an article in an article browsing interface;

[0049] displaying a material set in at least one blank area, wherein the at least one blank area is an area in the article browsing interface which is not used for displaying the part of content, and recommended materials in the material set are derived from images used for information recommendation;

[0050] in response to a zoom operation, displaying a zoom result of the at least part of content in the article browsing interface, and displaying a new material set in at least one new blank area;

[0051] wherein the at least one new blank area is an area in the article browsing interface which is not used for displaying the zoom result, and materials in the new material set are derived from the images.

[0052] Embodiments of the present application provide an information recommendation device based on images, the device comprising:

[0053] a third display module configured to display at least part of content of an article in an article browsing interface;

[0054] the third display module is further configured to display a material set in at least one blank area, wherein the at least one blank area is an area in the article browsing interface which is not used for displaying the part of content, and recommended materials in the material set are derived from images used for information recommendation;

[0055] a zoom module configured to, in response to a zoom operation, display a zoom result of the at least part of content in the article browsing interface, and display a new material set in at least one new blank area; wherein the at least one new blank area is an area in the article browsing interface which is not used for displaying the zoom result, and materials in the new material set are derived from the images.

[0056] In the above solution, the third display module is further configured to, when a position of the new blank area in the article is the same as a position of the blank area in the article, perform any one of the following processing on the new blank area: synchronously zooming the material set displayed in the at least one blank area to obtain a new material set adapted to the at least one new blank area, and displaying the new material set in the at least one new blank area; adaptively adding or reducing materials in the material set displayed in the at least one blank area to obtain a new material set adapted to the at least one new blank area, and displaying the new material set in the at least one new blank area.

[0057] Embodiments of the present application provide an electronic device, the electronic device comprising:

[0058] a memory for storing executable instructions;

[0059] a processor for executing the executable instructions stored in the memory to implement the image-based information recommendation method provided by the embodiments of the present application.

[0060] The embodiments of the present application provide a computer readable storage medium storing executable instructions for being executed by a processor to implement the image-based information recommendation method provided by the embodiments of the present application.

[0061] The embodiments of the present application provide a computer program product comprising a computer program or instructions, which are executed by a processor to implement the image-based information recommendation method provided by the embodiments of the present application.

[0062] The embodiments of the present application have the following beneficial effects:

[0063] The image used for information recommendation is subjected to material identification processing to obtain a plurality of recommendation materials, the recommendation materials are selected in descending order of priority, the selected recommendation materials are used to form a material set corresponding to a blank area in an information browsing interface, the material set corresponding to the blank area is displayed in the blank area, so that the image can be displayed in the blank area of the information browsing interface, the display resource utilization rate of the information browsing interface and the recommendation flexibility of information recommendation are effectively improved, and the recommendation efficiency of information recommendation is improved due to the priority display of the recommendation materials with higher priority. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 is a structural schematic diagram of an image-based information recommendation system architecture provided by the embodiments of the present application;

[0065] Figures 2A-2B is a structural schematic diagram of an electronic device provided by the embodiments of the present application;

[0066] Figures 3A-3D is a flow schematic diagram of an image-based information recommendation method provided by the embodiments of the present application;

[0067] Figures 4A-4B is an interface schematic diagram of an image-based information recommendation method provided by the embodiments of the present application;

[0068] Figures 5A-5B is an interface schematic diagram of an image-based information recommendation method provided by the embodiments of the present application;

[0069] Figures 6A-6B is an interface schematic diagram of an image-based information recommendation method provided by the embodiments of the present application;

[0070] Figures 7A-7B is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application;

[0071] Figure 8 is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application;

[0072] Figure 9 is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be described in further detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of the present application.

[0074] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0075] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0077] Before the embodiments of the present application are described in further detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0078] 1) Recommended material: the recommended material is a part of the image that can independently output information, taking an advertising image as an example, the recommended material is the title, product image, price, etc. in the advertising image.

[0079] 2) Recommended Target: This refers to the target of the recommendation. Since the information is presented via a terminal, the target of the recommendation process is the user operating the terminal. Therefore, the terms "target" and "user" are used interchangeably in the following text. It is understood that the user here can be a natural person operating a terminal or a robot program running on the terminal that can simulate a human.

[0080] In related technologies, images used for information recommendation (such as advertising images) are usually placed at fixed exposure positions. The exposure positions will adopt their own fixed sizes in different scenarios. Therefore, the display of recommended images is highly dependent on the scene and exposure position size. In related technologies, the placement of recommended images is not flexible enough and can only be placed as a whole, which can easily lead to image exposure rate and recommendation efficiency.

[0081] The embodiments of the present application provide an image-based information recommendation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which are capable of displaying recommended materials in an image in a blank area of ​​an information browsing interface, thereby improving display resource utilization and recommendation flexibility. The following describes exemplary applications of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be a server. The following describes exemplary applications when the electronic device is implemented as a server.

[0082] See also Figure 1 , Figure 1 This is a structural diagram of an image-based information recommendation system provided by an embodiment of the present application. To support a novel application, the image-based information recommendation method provided by an embodiment of the present application can be implemented based on the collaboration of a server and a terminal. The terminal 400 is connected to the server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The terminal 400 sends an information browsing interface to the server 200. The server 200 generates a preview interface image that displays the novel in the information browsing interface, performs blank area recognition processing on the preview interface image to obtain multiple blank areas in the information browsing interface, performs material recognition processing on the image used for information recommendation to obtain multiple recommended materials in the image, obtains the priority of each recommended material, selects recommended materials in order from high to low priority, and forms a material set corresponding to the blank area based on the selected recommended materials. The server 200 sends the material set to the terminal and displays the material set in the blank area of ​​the terminal's information browsing interface.

[0083] In some embodiments, the image-based information recommendation method provided by the embodiments of the present application can be implemented based on a terminal alone. The terminal performs blank area identification processing on an information browsing interface including at least one piece of information, to obtain a plurality of blank areas in the information browsing interface; performs material identification processing on an image used for information recommendation, to obtain a plurality of recommendation materials in the image; obtains a priority of each recommendation material; selects the recommendation materials in order from high to low according to the priority, and displays a material set corresponding to the blank area in at least one blank area of the information browsing interface based on the selected recommendation materials.

[0084] In some embodiments, the server 200 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0085] In some embodiments, the terminal or the server can implement the image-based information recommendation method provided by the embodiments of the present application by running a computer program. For example, the computer program can be a native program or a software module in an operating system; can be a native application program (APP), i.e., a program that needs to be installed in an operating system to run, such as a news APP or an e-commerce APP; can be a mini program, i.e., a program that only needs to be downloaded into a browser environment to run; or can be a mini program that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in.

[0086] Referring to Figure 2A , Figure 2A is a structural schematic diagram of an electronic device provided by the embodiments of the present application, taken as a server 400, Figure 2A The server 400 shown in FIG. 4 includes at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 420. It can be understood that the bus system 420 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 420 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, only the data bus is shown in Figure 2AThe various buses are all interconnected, along with some other full-duplex buses that are part of the bus system 440.

[0087] The processor 410 can be an integrated circuit chip that has the processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0088] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0089] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 optionally includes one or more storage devices remotely located from the processor 410 in physical location.

[0090] The memory 450 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), and volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0091] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0092] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0093] The network communication module 452 is used to communicate with other computing devices via one or more (wired or wireless) network interfaces 420, and exemplary network interfaces 420 include Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USA, Universal Serial Aus), etc.

[0094] a presentation module 453 for enabling presentation of information (e.g., a user interface for operating the peripheral device and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430;

[0095] an input processing module 454 for detecting and translating one or more user inputs or interactions from one or more input devices 432.

[0096] In some embodiments, the image-based information recommendation apparatus provided by the embodiments of the present application can be implemented in software, Figure 2A An image-based information recommendation apparatus 455-1 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a first region module 4551, a first material module 4552, a first priority module 4553, a first composition module 4554, and a first display module 4555, Figure 2A An image-based information recommendation apparatus 455-2 stored in the memory 450 is also shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a second display module 4556, Figure 2A An image-based information recommendation apparatus 455-3 stored in the memory 450 is also shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a third display module 4557 and a scaling module 4558. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0097] Referring to Figure 2B , Figure 2B is a structural schematic diagram of an electronic device provided by the embodiments of the present application, taken as a server 200 for example, Figure 2B The server 200 shown includes at least one processor 210, a memory 250, and at least one network interface 220. The various components in the terminal 200 are coupled together by a bus system 220. It can be understood that the bus system 220 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 220 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity in Figure 2B , all kinds of buses are marked as the bus system 240.

[0098] The processor 210 can be an integrated circuit chip that has a processing capability of signals, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general purpose processor can be a microprocessor or any conventional processor.

[0099] The memory 250 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical drives, etc. The memory 250 optionally includes one or more storage devices remotely located from the processor 210.

[0100] The memory 250 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0101] In some embodiments, the memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0102] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks.

[0103] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including Bluetooth, wireless fidelity (WiFi), and universal serial bus (USB), etc.

[0104] In some embodiments, the image-based information recommendation apparatus provided by the embodiments of the present application can be realized in a software manner, Figure 2B An image-based information recommendation apparatus 255 stored in the memory 250 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a second region module 2551, a second material module 2552, a second priority module 2553, a second composition module 2554, and a sending module 2555, which are logical, and thus can be combined or further split according to the implemented functions, and the functions of each module will be described below.

[0105] The exemplary application and implementation of the terminal provided by the embodiments of the present application will be described to illustrate the image-based information recommendation method provided by the embodiments of the present application.

[0106] Referring to Figure 3A , Figure 3A is a flowchart of the image-based information recommendation method provided by the embodiments of the present application, which will be described in combination with Figure 3A steps 101-105 shown in the figure.

[0107] In step 101, blank area identification processing is performed on an information browsing interface including at least one information, to obtain a plurality of blank areas in the information browsing interface.

[0108] As an example, the information browsing interface can be an information stream interface for displaying an information stream, for example, an information stream including a plurality of information, can be a conversation interface for displaying conversation messages, can be a webpage interface for displaying an article (displaying specific content of one information), and the like.

[0109] In some embodiments, the blank area identification processing on the information browsing interface including at least one information in step 101 to obtain a plurality of blank areas in the information browsing interface can be implemented by the following technical solution: obtaining a preview interface image of the information browsing interface, the preview interface image including at least one information, the preview interface image being an interface image displaying information content of the at least one information in the information browsing interface; identifying pixels in the preview interface image that do not belong to the information content as background pixels; performing connected processing on the mutually adjacent background pixels of the plurality of background pixels to obtain a plurality of connected areas, and performing cropping processing on each connected area to obtain a plurality of blank areas conforming to a preset shape.

[0110] As an example, the blank area identification processing can be performed on the information browsing interface that has already displayed at least one information, or the preview interface image of the information browsing interface is obtained, the preview interface image being a preview image obtained by previewing at least one information in the information browsing interface, and the blank area identification processing is performed on the preview interface image, and the at least one information is not displayed in the information browsing interface when the blank area identification processing is performed.

[0111] As an example, a preview interface image of the information browsing interface is acquired, the preview interface image is an interface image in which information content is displayed in the information browsing interface, the client detects the information browsing interface, calculates the information content to be browsed by the user according to the information browsing interface and the model, acquires the preview interface image in which the information content is displayed in the information browsing interface, identifies pixels in the preview interface image that do not belong to the information content as background pixels, filters out pixels of the information content and pixels of the background, performs connected processing on background pixels adjacent to each other in the plurality of background pixels, obtains a plurality of connected regions, since the connected regions are obtained by performing the connected processing on the background pixels adjacent to each other, the connected regions are irregular, and even some connected regions further include regions between words, therefore, the connected regions need to be cropped, the purpose of the cropping is to acquire a blank region in the connected region that meets a preset shape, each connected region is cropped to obtain a plurality of blank regions that meet the preset shape, for example, a rectangular region is cropped from the connected region, or a trapezoidal region is cropped from the connected region, or a ladder-shaped region is cropped from the connected region, after cropping a certain connected region, one or more blank regions that meet the preset shape are obtained. Through the embodiment of the present application, the blank region can be automatically identified, and therefore, the identification of the blank region can be accurately and efficiently implemented.

[0112] In some embodiments, the above-mentioned identification of the pixels in the preview interface image that do not belong to the information content as background pixels can be implemented by the following technical solutions: acquiring a standard binarization result corresponding to the blank region; performing binarization processing on each pixel in the preview interface image to obtain a binarization result of each pixel; performing compensation processing on the binarization result of each pixel to obtain a compensated binarization result of each pixel; when the compensated binarization result is the same as the standard binarization result, identifying the pixel corresponding to the compensated binarization result as a background pixel.

[0113] As an example, the standard binarization result corresponding to the blank area is obtained. Binarization is to merge the colors of pixels at both ends, that is, to map the color value of each pixel in the image to any one of two values, for example, to map the color value to 0 or 1; for another example, to map the color value to 0 or 255. A mapping relationship needs to be configured between binarization, that is, the standard binarization result of the blank area is obtained. For example, if the standard binarization result is 1, the pixel with a subsequent binarization result of 1 is a pixel of the blank area. Each pixel in the preview interface image is subjected to binarization processing to obtain a binarization result of each pixel. When the standard binarization result of the blank area is 1, the binarization result of the pixel with a color value greater than a set threshold is determined to be 1, and the binarization result of the pixel with a color value not greater than the set threshold is determined to be 0. An image is composed of each pixel point. Each pixel point can be quantized into red, yellow and blue color values. According to the red, yellow and blue color values, the set threshold values corresponding to the three colors are determined. Based on the set threshold value of each color, the pixel can be mapped to 0 or 1 as the binarization result. The binarization result of each pixel is compensated to obtain a compensated binarization result of each pixel. An image often has noise points. These noise points are usually single points. Sometimes, several single pixel points are connected to form an interference line. When denoising, the noise points and the interference line need to be removed. The values of the four pixel points around each pixel point can be taken out. If the binarization result of more than two pixel points around the pixel point is 1, it is considered that the pixel point is a noise point, and the pixel point can be set as a background pixel point with a binarization result of 1. When performing binarization processing, a pixel point with a large color value in a character can be filtered as a background. At this time, the pixel point needs to be compensated. Similarly, the four characters around the pixel point are counted. If the binarization result of more than two pixel points around the four pixel points is 0, it is considered that the pixel point is also a character pixel point with a binarization result of 0. When the compensated binarization result is the same as the standard binarization result, the pixel corresponding to the compensated binarization result is recognized as a background pixel.

[0114] In some embodiments, the above-mentioned obtaining the standard binarization result corresponding to the blank area can be realized by the following technical solutions: determining an information color value of a representative pixel belonging to information content and a blank color value of a representative pixel not belonging to the information content; when the blank color value is less than the information color value, taking a zero value as the standard binarization result of the blank area; and when the blank color value is not less than the information color value, taking a non-zero value as the standard binarization result of the blank area. The standard binarization result of the blank area can be determined by the manner provided in the embodiments of the present application, so that the pixel of the information content is not recognized as the pixel of the blank area when the background and the information content are separated subsequently, and the accuracy of blank area recognition is improved.

[0115] As an example, the information color value of the representative pixel belonging to the information content and the blank color value of the representative pixel not belonging to the information content are determined, the representative pixel belonging to the information content can be any pixel of the information content, the information color value, i.e., the color value of the representative pixel of the information content, includes a green value, a red value and a blue value, the representative pixel not belonging to the information content can be any pixel not belonging to the information content, the blank color value, i.e., the color value of the representative pixel not belonging to the information content, includes a green value, a red value and a blue value, when the blank color value is less than the information color value, any one color value of the background pixel is less than the corresponding type of color value of the pixel of the information content, and therefore zero value is taken as the standard binarization result of the blank area, when the blank color value is not less than the information color value, all types of color values of the background pixel are not less than the color value of the pixel of the information content, and therefore non-zero value is taken as the standard binarization result of the blank area.

[0116] In step 102, material recognition processing is performed on the image for information recommendation, and a plurality of recommendation materials in the image are obtained.

[0117] In some embodiments, any one of the following processing is performed on the plurality of candidate images: determining the relevance of the candidate image to the information content displayed in the information browsing interface, and taking the candidate image with the highest relevance as the image for information recommendation; determining the interest degree of the to-be-recommended object to the candidate image, and taking the candidate image with the highest interest degree as the image. Through the embodiments of the present application, the recommendation accuracy and the recommendation efficiency of the image for information recommendation can be improved.

[0118] As an example, the candidate image can be a plurality of images issued by an advertiser and requiring information recommendation, the relevance of the candidate image to the information content displayed in the information browsing interface is determined, the candidate image with the highest relevance is taken as the image for information recommendation, the content of the candidate image is recognized through image understanding technology, and an image keyword related to the content of the candidate image is generated, a text keyword of the information content is recognized through semantic recognition technology, the relevance of the image keyword and the text keyword is determined as the relevance of the candidate image to the information content displayed in the information browsing interface, for example, the information content relates to clothing design or a certain brand, and the image for information recommendation can be the candidate image with the highest relevance to the brand, the interest degree of the to-be-recommended object to the candidate image is determined, and the candidate image with the highest interest degree is taken as the image, the to-be-recommended object is a user receiving information through the information browsing interface using a client, object data of the to-be-recommended object is processed for feature extraction through a neural network model to obtain object features, the object data includes historical behavior data such as browsing data, collection data, and like data, and the object features can be used to express the interest of the to-be-recommended object, in addition, image semantic features are obtained by performing image feature extraction on the candidate image through the neural network model, and the similarity between the object features and the image semantic features is determined as the interest degree of the to-be-recommended object to the candidate image.

[0119] In some embodiments, the material recognition processing on the image for information recommendation in step 102 obtains a plurality of recommendation materials in the image, which can be implemented through the following technical solution: performing preprocessing on the image to obtain a preprocessed image, wherein the preprocessing includes at least one of the following: image denoising processing, image smoothing processing, and image transformation processing; performing feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; performing mapping processing on the image features to obtain probabilities of a plurality of material types corresponding to each sampling pixel block in the image; and performing the following processing for each sampling pixel block: determining the material type with the highest probability as the material type corresponding to the sampling pixel block; and performing merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain a recommendation material corresponding to the same material type. Through the embodiments of the present application, the recommendation materials can be recognized and classified, thereby realizing effective segmentation of the image and facilitating flexible display of the image subsequently.

[0120] As an example, let's use an advertisement image as an example. The image is preprocessed. Preprocessing primarily involves image processing operations such as denoising, smoothing, and transformation, which can enhance key features of the image. Next, feature extraction and selection are performed on the preprocessed image to obtain image features. A classifier performs joint array-based recognition and classification to determine the material type of each pixel block. Material types include: main title / product selling point, subtitle / product benefit point, product image, button element, advertiser logo, and background image. Adjacent sampled pixel blocks of the same material type are merged to obtain recommended materials corresponding to the same material type. For example, if pixel blocks 1, 2, and 3 are adjacent and all represent the main title, then pixel blocks 1, 2, and 3 are merged to obtain a recommended material with the main title as the material type.

[0121] In some embodiments, the above-mentioned feature extraction processing of the preprocessed image to obtain image features of the preprocessed image can be implemented by the following technical solutions: performing feature extraction processing on the preprocessed image to obtain three-dimensional features for each pixel block in the preprocessed image; based on the spatial features in the three-dimensional features of each pixel block, performing spatial sampling processing on multiple pixel blocks in the preprocessed image to obtain multiple sampled pixel blocks; and splicing the three-dimensional features of the multiple sampled pixel blocks into image features. Through the embodiments of the present application, image features used to characterize relatively important parts of the image can be obtained, thereby reducing the computational complexity of the processing process.

[0122] As an example, let's take the case of an advertisement image as an example. Feature extraction and selection are performed on the preprocessed advertisement image. The implementation of a convolutional neural network actually includes convolutional and pooling layers. The convolutional layer breaks the advertisement image into multiple 3*3 or 5*5 pixel blocks, using three numbers to represent the content of each area in the advertisement image, representing height, width, and color respectively. These output values ​​are then arranged in a graph group to obtain a three-dimensional numerical representation (three-dimensional feature) of each pixel block. Based on the spatial features in the three-dimensional features of each pixel block, multiple pixel blocks in the preprocessed image are spatially sampled to obtain multiple sampled pixel blocks. The pooling layer combines the spatial dimensions of this three-dimensional (or four-dimensional) graph group with the sampling function to output a joint array (image feature) that only contains the relatively important parts of the image.

[0123] In step 103, the priority of each recommended material is obtained.

[0124] In some embodiments, the priority of each recommended material in step 103 can be obtained by the following technical solutions: performing any one of the following processes: obtaining a priority configured in advance for each material type, and determining the priority of the material type of the recommended material as the priority of the recommended material; obtaining a feature parameter of the recommended material, and determining a priority positively correlated with the feature parameter, wherein the feature parameter includes at least one of the following: an area of the recommended material, and a degree of interest of the object to be recommended in the recommended material.

[0125] As an example, the image for information recommendation includes a plurality of recommended materials, and the recommended materials have different priorities, and the recommended materials with higher priorities are preferentially displayed when the recommended materials are displayed.

[0126] In step 104, the recommended materials are selected in a descending order of priority, and a material set corresponding to the blank area is formed based on the selected recommended materials.

[0127] As an example, the sum of the sizes of at least one recommended material included in the material set is adapted to the size of the blank area, and the adaptation represents that the at least one recommended material can be displayed in the blank area, that is, the sum of the areas of the at least one recommended material is not greater than the area of the blank area, and the ratio difference between the aspect ratio of the composition result of the at least one recommended material and the aspect ratio of the blank area is less than a ratio difference threshold.

[0128] In some embodiments, referring to Figure 3B , Figure 3B is a flowchart of the information recommendation method based on an image provided by the embodiments of the present application, and the recommended materials are selected in a descending order of priority in step 104 based on the selected recommended materials and the blank area, which can be implemented by Figure 3B steps 1041 to 1044 shown in the figure.

[0129] In step 1041, at least one blank area meeting a blank area condition is obtained from a plurality of blank areas, and is taken as a blank area to be filled.

[0130] In some embodiments, the blank area condition includes at least one of the following: the size of the blank area is greater than a size threshold; the distance between the position of the blank area and the center of the information browsing interface is not less than a first distance threshold, that is, the blank area is in a relatively central position in the information browsing interface, so as to improve the recommendation efficiency of the recommended materials, and the minimum distance between the blank area and the displayed material set is greater than a second distance threshold, that is, the recommended materials for information recommendation are avoided from being frequently displayed, so as to improve the user experience.

[0131] As an example, the displayed recommended material can be determined for each blank area, or the blank areas are filtered, and the recommended material is displayed in the blank areas meeting the blank area condition, for example, blank areas with a size less than a size threshold are filtered out, and the size threshold can be the average of the minimum size of the sample blank areas in which the recommended material is displayed in the plurality of sample images.

[0132] The following steps 1042 to 1044 are performed for each blank area to be filled:

[0133] In step 1042, the number of materials corresponding to the size of the blank area is queried in the mapping table;

[0134] In step 1043, the plurality of recommended materials are sorted in descending order of priority, and at least one recommended material meeting the number of materials is obtained in the front of the sorting;

[0135] In step 1044, the at least one recommended material is scaled based on the size of the blank area, and the scaled at least one recommended material forms a material set.

[0136] As an example, when the size of the blank area is greater than 750*300 pixels, all recommended materials of all priorities are displayed, when 450*300 pixels is less than the size of the blank area, and the size of the blank area is less than 750*300 pixels, the top three recommended materials of the priority are displayed; when 200*300 pixels is less than the size of the blank area, and the size of the blank area is less than 450*300 pixels, the top two recommended materials of the priority are displayed; when the size of the blank area is less than 200*300 pixels, no recommended material is displayed, because the size of the blank area is too small, and the user cannot clearly see the displayed recommended material. Through the embodiment of the present application, the flexible display of the recommended material can be realized, thereby effectively improving the recommendation efficiency of the recommended material.

[0137] In some embodiments, the recommended material is selected in step 104 in order of priority from high to low, and based on the selected recommended material group and the blank area, the following technical solutions can be used: at least one blank area meeting the blank area condition is obtained from the plurality of blank areas, and is taken as a to-be-filled blank area; for each to-be-filled blank area, the following processing is performed: the plurality of recommended materials are sorted in order of priority from high to low, and for each recommended material, the following processing is performed according to the sorting result: when the fillable size of the to-be-filled blank area is not less than the size of the recommended material, the recommended material is taken as a target material for filling the to-be-filled blank area; and when the fillable size of the to-be-filled blank area is less than the size of the recommended material, a material set corresponding to the blank area is generated based on the already selected target material. Through the embodiments of the present application, flexible display of the recommended material can be realized, and the recommended material with a higher priority is preferentially displayed, thereby effectively improving the recommendation efficiency of the recommended material.

[0138] As an example, at least one blank area meeting the blank area condition is obtained from the plurality of blank areas, and is taken as a to-be-filled blank area; for each blank area, the displayed recommended material can be determined, or the blank area is filtered, and the recommended material is displayed in the blank area meeting the blank area condition, for example, blank areas with a size less than a size threshold are filtered out, and the size threshold can be the average of the minimum size of the sample blank area in which the recommended material is displayed in the plurality of sample images.

[0139] As an example, taking an image including five recommended materials as an example, the priority of the recommended material A is higher than that of the recommended material B, the priority of the recommended material B is higher than that of the recommended material C, and so on, for each to-be-filled blank area, the following processing is performed: the recommended material A is taken as a target material for filling the to-be-filled blank area 1, then it is judged whether the fillable blank area of the to-be-filled blank area 1 can fill the recommended material B, when the fillable size of the to-be-filled blank area is not less than the size of the recommended material B, the recommended material B is continuously taken as a target material for filling the to-be-filled blank area 1, then it is judged whether the fillable blank area of the to-be-filled blank area 1 can fill the recommended material C, and when the fillable size of the to-be-filled blank area is less than the size of the recommended material C, a material set corresponding to the blank area is generated based on the already selected target material A and the target material B.

[0140] In step 105, the material set corresponding to the blank area is displayed in at least one blank area of the information browsing interface.

[0141] As an example, the material set can be displayed in the blank area after the information content is displayed in the information browsing interface, or the material set can be displayed in the blank area while the information content is displayed in the information browsing interface, i.e., the blank area in which the recommended material can be displayed is identified in advance in the case that the information browsing interface is filled with information content.

[0142] In some embodiments, in the at least one blank area of the information browsing interface, at least one information is displayed in the information browsing interface while the material set corresponding to the blank area is displayed in the blank area, the at least one blank area is actually an area in the information browsing interface which is not used to display the at least one information, when the information browsing interface is an article browsing interface, at least part of the content of an article is displayed in the article browsing interface, the article can be a novel, a log, a blog, etc., and the article browsing interface can be a web interface or a mobile phone interface.

[0143] In some embodiments, when the information browsing interface is an article browsing interface, a zoom result of at least part of the content of an article is displayed in the article browsing interface in response to a zoom operation.

[0144] For example, the scaling operation can be a scaling operation of the article in a same proportion, such as a scaling operation of a page display proportion, and the object of the scaling operation only includes the original content of the article, while for the recommended material, since the recommended material can be displayed in the form of a floating layer embedded in the blank area, the recommended material can be adaptively changed according to the blank area formed after the scaling operation; for another example, the scaling operation can be a scaling operation of the text alone, such as changing the font size of the text; for yet another example, the scaling operation can be a scaling operation of the line spacing. In general, the scaling operation is an operation of adjusting the layout of the article content, the blank area identification processing is performed on the article browsing interface including the scaling result, a plurality of new blank areas are obtained, a new material set for displaying in the new blank area is acquired, when the position of the new blank area in the article is the same as the position of the blank area in the article, the new material set corresponding to the material set of the blank area is acquired, the sum of the sizes of at least one recommended material included in the new material set is adapted to the size of the new blank area, for example, the material set includes recommended material A and recommended material B, the new material set also includes recommended material A and recommended material B, and the sizes of recommended material A and recommended material B are adjusted to adapt to the size of the new blank area, and the recommended material can also be selected in order of priority from high to low, and the selected recommended material is used to form the material set corresponding to the new blank area, and the specific implementation manner can be referred to step 104, and the blank area is replaced by the new blank area. In order to ensure that the new material set is adapted to the new blank area, the material set can also be adaptively increased or decreased to obtain a new material set adapted to the new blank area. After the new material set for displaying in the new blank area is acquired, the new material set is displayed in at least one new blank area.

[0145] Through the embodiments of the present application, the new material set can be adaptively acquired and displayed according to the change of the blank area during the article reading process, so that the recommended material can be efficiently and flexibly displayed during the user reading process, and the display resource utilization rate of the article browsing interface and the recommendation flexibility of the information recommendation are effectively improved.

[0146] In some embodiments, referring to Figure 3C , Figure 3C is a flowchart of an information recommendation method based on an image provided by the embodiments of the present application, which will be described in combination with steps 201-207 shown in Figure 3C .

[0147] In step 201, the terminal sends an information browsing interface to the server.

[0148] As an example, the terminal sends the information browsing interface with information content to the server, so that the server performs the blank area identification processing, or the terminal only sends the information browsing interface to the server, and the server fills the information content to be displayed into the information browsing interface, forms a preview interface image, and performs the blank area identification processing on the preview interface image.

[0149] In step 202, the information browsing interface in the terminal is subjected to blank area identification processing, and a plurality of blank areas in the information browsing interface are obtained.

[0150] In step 203, material identification processing is performed on the image for information recommendation, and a plurality of recommendation materials in the image are obtained.

[0151] In step 204, the priority of each recommendation material is obtained.

[0152] In step 205, the recommendation materials are selected in descending order of priority, and a material set corresponding to the blank area is formed based on the selected recommendation materials.

[0153] As an example, the sum of the sizes of at least one recommendation material included in the material set is adapted to the size of the blank area.

[0154] The embodiments of steps 202 to 205 can refer to steps 101 to 104.

[0155] In step 206, the terminal is sent the material set corresponding to at least one blank area.

[0156] In step 207, the material set corresponding to the blank area is displayed in at least one blank area of the information browsing interface.

[0157] As an example, the material set can be displayed in the blank area after the information content is displayed in the information browsing interface, or the material set can be displayed in the blank area while the information content is displayed in the information browsing interface, that is, the blank area in which the recommendation material display can be performed when the information browsing interface is filled with information content is identified in advance.

[0158] Through the embodiments of the present application, the server can identify the blank area and match the material set adapted to the blank area, so that the computing processing pressure of the terminal can be reduced, and the overall processing speed can be improved, thereby improving the user experience.

[0159] In some embodiments, when the information browsing interface is an article browsing interface, in response to the zoom operation, the at least part of the content of the article displayed in the article browsing interface is zoomed.

[0160] In some embodiments, when the information browsing interface is an article browsing interface, in response to the zoom operation, the at least part of the content of the article displayed in the article browsing interface is zoomed.

[0161] For example, the zoom operation can be a zoom operation for the article, such as a zoom operation for the page display scale. The object of the zoom operation only includes the original content of the article, and for the recommended material, since the recommended material can be displayed in the blank area in the form of a floating layer, the recommended material can be adaptively changed according to the blank area formed after zooming. For another example, the zoom operation can be a zoom operation for the text only, such as changing the font size of the text. For another example, the zoom operation can be a zoom operation for the line spacing. In general, the zoom operation is an operation of adjusting the layout of the article content. The terminal sends the zoom result to the server, and the server performs the following processing: performing blank area identification processing on the article browsing interface including the zoom result to obtain a plurality of new blank areas, obtaining a new material set for displaying in the new blank area, when the position of the new blank area in the article is the same as the position of the blank area in the article, obtaining a new material set corresponding to the material set of the blank area, the sum of the sizes of at least one recommended material included in the new material set is adapted to the size of the new blank area, for example, the material set includes recommended material A and recommended material B, the new material set also includes recommended material A and recommended material B, and the sizes of recommended material A and recommended material B are adjusted to adapt to the size of the new blank area, and the recommended materials can also be selected in order of priority from high to low, and the selected recommended materials are used to form the material set corresponding to the new blank area. The specific implementation can be referred to step 205, and the blank area is replaced by the new blank area. In order to ensure that the new material set is adapted to the new blank area, the material set can also be adaptively increased or decreased to obtain a new material set adapted to the new blank area. After obtaining the new material set for displaying in the new blank area, the new material set is sent to the terminal, and the terminal displays the new material set in the at least one new blank area.

[0162] Through the embodiment of the present application, new material set can be adaptively acquired and displayed according to the change of the blank area in the article reading process, so that the recommended material can be efficiently and flexibly displayed in the user reading process, and the display resource utilization rate of the article browsing interface and the recommendation flexibility of information recommendation are effectively improved.

[0163] In some embodiments, referring to Figure 3D , Figure 3D is a flowchart of the image-based information recommendation method provided by the embodiment of the present application, which will be described in combination with Figure 3D steps 301-302 shown in the figure

[0164] In step 301, at least one information is displayed in an information browsing interface.

[0165] In step 302, a material set is displayed in at least one blank area remaining after the display of the at least one information.

[0166] As an example, the material set includes at least one recommended material, and is obtained by identifying an image, the sum of the sizes of the at least one recommended material included in the material set is adapted to the size of the blank area, and the recommended material is obtained in the order from high to low according to the priority of the multiple recommended materials in the image.

[0167] In some embodiments, at least one information is displayed in an information browsing interface; a material set is displayed in at least one blank area, the at least one blank area is between paragraphs of the at least one information; wherein the material set includes at least one recommended material, the recommended material is derived from an image used for information recommendation, and the sum of the sizes of the at least one recommended material included in the material set is adapted to the size of the blank area.

[0168] As an example, the blank area is naturally formed based on the layout and typesetting of the at least one information in the information browsing interface, and the blank area can be between information paragraphs, referring to Figure 6B and Figure 7B , Figure 6B In the blank area in the blank area is a blank area formed around several paragraphs, Figure 7B the blank area in the blank area is a blank area formed around the last paragraph and the bottom of the page. Since the recommended material is displayed by using the blank area between the paragraphs of the information, the user's attention to the recommended material can be higher, thereby improving the recommendation efficiency of the recommended material.

[0169] In some embodiments, at least part of the content of the article is displayed in an article browsing interface; a set of materials is displayed in at least one blank area, the at least one blank area being an area in the article browsing interface that is not used to display the part of the content, the recommended materials in the set of materials being derived from the image used for information recommendation, the sum of the sizes of the at least one recommended material included in the set of materials being equal to the sum of the sizes of the blank area; in response to a zoom operation, a zoom result of the at least part of the content is displayed in the article browsing interface, and a new set of materials is displayed in at least one new blank area; the at least one new blank area being an area in the article browsing interface that is not used to display the zoom result, the materials in the new set of materials being derived from the image.

[0170] In some embodiments, the above-mentioned displaying of the new set of materials in the at least one new blank area can be implemented by the following technical solutions: when the position of the new blank area in the article is the same as the position of the blank area in the article, the following any one of the processing is performed on the new blank area: performing synchronous zoom processing on the set of materials displayed in the at least one blank area to obtain a new set of materials adapted to the at least one new blank area, and displaying the new set of materials in the at least one new blank area; performing adaptive material addition and reduction processing on the set of materials displayed in the at least one blank area to obtain a new set of materials adapted to the at least one new blank area, and displaying the new set of materials in the at least one new blank area. The adaptation means that the sum of the sizes of the recommended materials included in the new set of materials is equal to the sum of the sizes of the at least one new blank area.

[0171] Referring to Figure 9 , Figure 9 is an interface schematic diagram of the image-based information recommendation method provided by the embodiments of the present application, at least part of the content of the article is displayed in an article browsing interface 901, a set of materials 902 is displayed in at least one blank area, in response to a zoom operation, a zoom result of the at least part of the content of the article is displayed in the article browsing interface, and a new set of materials 903 is displayed in at least one new blank area.

[0172] As an example, in response to the zoom operation, the zoom result of at least part of the content of the article is displayed in the article browsing interface, for example, the zoom operation can be a zoom operation of the article in a same scale, for example, a zoom operation of the page display scale in a same scale, the object of the zoom operation in a same scale only includes the original content of the article, and for the recommended material, since the blank area can be displayed by embedding in the form of a floating layer, the recommended material can be adaptively changed according to the blank area formed after zooming in a same scale; for another example, the zoom operation can be a zoom operation of the text alone, for example, changing the font size of the text; for yet another example, the zoom operation can be a zoom operation of the line spacing. In general, the zoom operation is an operation of adjusting the layout of the article content, the article browsing interface including the zoom result is subjected to blank area identification processing to obtain a plurality of new blank areas, a new material set for display in the new blank area is obtained, when the position of the new blank area in the article is the same as the position of the blank area in the article, a new material set corresponding to the material set of the blank area is obtained, the new material set corresponding to the material set of the blank area is displayed in the new blank area, the sum of the sizes of at least one recommended material included in the new material set is adapted to the size of the new blank area, for example, the material set includes recommended material A and recommended material B, the new material set also includes recommended material A and recommended material B, and the sizes of recommended material A and recommended material B are adjusted to adapt to the size of the new blank area, and the recommended material can also be selected in order of priority from high to low, and the selected recommended material is used to form the material set corresponding to the new blank area, and the specific implementation manner can refer to step 104, and the blank area is replaced by the new blank area. After obtaining the new material set for display in the new blank area, the new material set is displayed in at least one new blank area.

[0173] Through the embodiments of the present application, the new material set can be adaptively obtained and displayed according to the change of the blank area during the article reading process, so that the recommended material can be efficiently and flexibly displayed during the user reading process, and the display resource utilization rate of the article browsing interface and the recommendation flexibility of information recommendation are effectively improved.

[0174] In the following, an example application of the embodiments of the present application in an actual application scenario will be described.

[0175] The image-based information recommendation method provided in the embodiment of the present application can be implemented based on the collaboration of a server and a terminal. The terminal sends an information browsing interface to the server, the server generates a preview interface image that displays the novel in the information browsing interface, performs blank area recognition processing on the preview interface image, and obtains multiple blank areas in the information browsing interface. The image used for information recommendation is subjected to material recognition processing to obtain multiple recommended materials in the image, obtains the priority of each recommended material, selects recommended materials in order from high to low priority, and forms a material set corresponding to the blank area based on the selected recommended materials. The server sends the material set to the terminal and displays the material set in the blank area of ​​the information browsing interface of the terminal.

[0176] The image-based information recommendation method provided by the embodiments of this application uses AI image recognition and content recognition to segment the content of the advertisement image and define priorities. It then automatically combines and intelligently delivers creative materials based on the size and shape of the blank space within the novel text. When it detects that the size of the blank space on the page meets a predetermined threshold, the combined ad creative is delivered. This improves page layout utilization, increases ad exposure, and enhances recommendation efficiency.

[0177] The image-based information recommendation method provided in the embodiment of the present application is mainly used in the advertising delivery system. By identifying the effective space on the current page of the terminal, advertising materials are intelligently delivered. While improving the layout utilization rate, it can also improve the exposure rate, consumption rate and conversion rate of the advertisement.

[0178] In some embodiments, see Figure 4A , Figure 4A This is a schematic diagram of the interface of the image-based information recommendation method provided in the embodiment of the present application. Figure 4A The advertisement image displayed in the browser is shown. The content of the advertisement image includes: advertiser logo, main title / product selling point 401, subtitle / product benefit point, product image, button elements, and background atmosphere image. Figure 4B , Figure 4B This is a schematic diagram of the interface of the image-based information recommendation method provided in the embodiment of the present application. Figure 4B The advertising materials of the identified advertising images are shown, and material priorities are determined based on the advertising materials. The material priorities are determined based on the actual advertising images delivered. The default order of material priorities is as follows: main title / product selling points, subtitle / product benefit points, product images, button elements, advertiser logos, and background atmosphere images. A priority label is bound to each recommended material. For example, the priority label of the main title is 1, and the priority label of the product image is 3. The smaller the priority label number, the higher the priority.

[0179] In some embodiments, see Figure 5A ,Figure 5A is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application, Figure 5A shows a reading page of a browser novel, and the reading page of the novel has more white spaces. See Figure 5B , Figure 5B is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application, Figure 5B shows an effective white space area 501 in the reading page of the novel identified, and the effective white space area refers to an area with a product of length and width (size) greater than 200 pixels*300 pixels. An area that is too small will cause the user to be unable to clearly see the displayed advertisement.

[0180] In some embodiments, see Figure 6A , Figure 6A is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application, Figure 6A shows a browser novel reading page, and an effective white space area for inserting an advertisement is intelligently generated. An advertisement picture intelligently generated based on advertisement materials is displayed in the effective white space area of the browser novel reading page. In order to ensure the display effect, the effective white space area needs to be a standard rectangle. See Figure 6B , Figure 6B is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application, Figure 6B shows an effect picture of an advertisement actually placed in a page. An advertisement picture is intelligently generated according to the size of the effective white space area and the priority of the advertisement materials, and the generated advertisement picture is inserted and displayed in the effective white space area of the page.

[0181] In some embodiments, see Figure 7A , Figure 7A is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application, Figure 7A shows a browser novel reading page, and an effective white space area for inserting an advertisement is intelligently generated. When it is detected that the size of the effective white space area is relatively small, advertisement materials with high priority will be preferentially displayed, and the advertisement materials will be simplified. See Figure 7B , Figure 7B is an interface schematic diagram of the image-based information recommendation method provided by the embodiment of the present application, Figure 7B shows the actual display effect of an effective white space area with a small size, and advertisement materials with high priority will be preferentially displayed.

[0182] In some embodiments, see Figure 8 , Figure 8Is the interface schematic diagram of the information recommendation method based on images provided by the embodiment of the application. In step 801, the server identifies the content of the advertisement graph and formulates the priority of each content. In step 802, the client detects the interface browsed by the user and returns to the server. In step 803, the server identifies the blank area of the interface. In step 804, the size of the blank area is generated. In step 805, the server selects the blank area with the largest size. In step 806, the corresponding advertisement graph is intelligently generated according to the blank area with the largest size in the page. In step 807, the server issues the corresponding advertisement graph to the client corresponding to the user identifier. In step 808, the client displays the intelligently generated advertisement graph.

[0183] In some embodiments, the server identifies the content of the advertisement graph and formulates the priority. The server performs image recognition on the advertisement graph uploaded by the advertiser, obtains each advertisement material in the advertisement graph, cuts each advertisement element from the advertisement graph, and formulates the material priority of the advertisement material. The advertisement material in the advertisement graph can be roughly divided into picture type material, text type material and button type material.

[0184] As an example, the scheme for identifying and processing the advertisement graph by AI technology and obtaining the material priority of each advertisement material is as follows: image recognition is mainly realized by a convolutional neural network. The advantage of the convolutional neural network is that it utilizes the principle that adjacent pixels in an image have strong correlation and strong similarity. Specifically, the correlation between two adjacent pixels in an image is stronger than that between two separate pixels in an image. The specific steps of image recognition are described below:

[0185] First, the sensor converts information such as light or sound into electrical information. Specifically, the advertisement graph is obtained and converted into information that can be processed by a machine.

[0186] Next, the advertisement graph is preprocessed. Preprocessing mainly refers to operations such as denoising, smoothing and transformation in image processing, which can enhance the important features of the advertisement graph.

[0187] Then, feature extraction and selection are performed on the preprocessed advertisement graph. In the implementation process of the convolutional neural network, it actually includes a convolutional layer and a pooling layer. The convolutional layer disperses the advertisement graph into multiple 3*3 or 5*5 pixel blocks. The three numbers represent the content of each region in the advertisement graph, representing height, width and color, respectively. Then, these output values are arranged in a graph group, and a three-dimensional numerical expression of each pixel block is obtained. The pooling layer combines the spatial dimension of this three-dimensional (or four-dimensional) graph group with a sampling function, and outputs a joint array containing only the relatively important part of the image.

[0188] Then, the recognition rule of the classifier is obtained by training, and the feature classification mapping mode can be obtained through the recognition rule, so as to form the related category label, and the classifier can realize a higher recognition rate. The image elements, text elements, button elements and the like can be obtained through the recognition and classification of the classifier, and more fine-grained recognition and classification processing is performed based on the image elements, text elements and button elements obtained through the recognition, so as to obtain the main title / product selling point, the sub-title / product benefit point, the product picture, the button element, the ad owner identifier and the background atmosphere picture. The main title / product selling point, the sub-title / product benefit point, the product picture, the button element, the ad owner identifier and the background atmosphere picture can also be directly obtained through the recognition and classification of the classifier.

[0189] The convolutional neural network can select a relatively small squeeze network. The squeeze network uses the network architecture of the FireModule to perform model compression. The FireModule first performs dimension reduction operation on the output of the previous layer (expand layer) by using 1x1 convolution to reduce the number of parameters, and then extracts features by using 1x1 convolution and 3x3 convolution in combination, fills the output dimensions after padding to be the same, and then splices. In this way, the model parameters can be effectively reduced, and the entire model size is only about 1.5M, and the running speed is fast. A large number of training samples are collected from the advertising scene, and finally the classifier is trained according to the training samples.

[0190] Finally, the material priority of the advertisement material is formulated according to the recognized advertisement material, and the material priority of the advertisement material is determined according to the actually launched advertisement picture. The default preset high-low order of the advertisement priority is as follows: main title / product selling point, sub-title / product benefit point, product picture, button element, ad owner identifier and background atmosphere picture. The server labels the recognized advertisement material with the priority.

[0191] In some embodiments, the client detects the information browsing interface of the mobile phone and returns to the server, the server identifies the blank area based on the image of the information browsing interface, and judges the size of the blank area. The client detects the information browsing interface, and returns the image of the information browsing interface, user identification, user model and other parameters to the server. The server calculates the page content to be browsed by the user in the future 5-20 pages according to the image and model of the information browsing interface, judges the size of the blank area after the information content of the Tian Cong information interface is browsed through the image and proportional positioning recognition method. The principle of judging the size of the blank area is as follows: the image is recognized by PHP, the recognition process includes binaryzation processing and noise compensation processing of the image, and finally the adjacent pixel blank area is provided, and the size of the largest blank area is judged.

[0192] Firstly, the image is composed of each pixel, each pixel can be quantified into red, yellow and blue three color values, according to the red, yellow and blue three color values, the threshold value corresponding to three colors is determined, based on the threshold value of each color, the background and the character can be separated, for example, if the color value of the background is higher than the threshold value, and the threshold value is higher than the color value of the character, the binary result of the background pixel can be determined as 1, and the binary result of the character pixel can be determined as 0.

[0193] Then, the image often has noise points, which are usually single points, sometimes several single pixels are connected into interference lines, and the noise points and interference lines need to be removed during noise reduction. The values of the four pixels around each pixel can be taken out, if the binary result of two or more pixels around the pixel is 1, then the pixel is considered to be a noise point, and the pixel can be set as a background pixel with a binary result of 1. When performing binary processing, the pixel with a large color value in the character may be filtered as background, at this time, the pixel needs to be compensated, and the four surrounding characters of the pixel are counted, if two or more pixels around the four pixels have a binary result of 0, then the pixel is also a character pixel with a binary result of 0.

[0194] Finally, based on the number of background pixels and character pixels, the area of the blank area composed of adjacent pixels is calculated, and the length-width product (size) of different blank areas in the information browsing interface is judged.

[0195] In some embodiments, the server selects the region with the largest size of the blank area, and generates a corresponding advertisement picture in the largest size blank area. The server calculates the largest size blank area according to the identified blank area, and selects the largest size blank area in the information browsing interface as the advertisement picture generation area. In order to improve the distribution efficiency and control the distribution frequency of the advertisement, and avoid the appearance of multiple advertisement pictures in the current page or three consecutive pages of the information browsing interface, the server calculates the largest size blank area of each page, and sets the advertisement exposure frequency. For example, when the 01 page appears an advertisement picture, the 02-04 pages do not appear an advertisement picture, and the advertisement exposure frequency can be adjusted dynamically according to the product.

[0196] As an example, when the server detects that the size of the advertisement image generation area is greater than 750 pixels * 300 pixels, all advertisement materials of the priority are displayed, when the size of the advertisement image generation area is less than 450 pixels * 300 pixels, the top three advertisement materials of the priority are displayed; when the size of the advertisement image generation area is less than 450 pixels * 300 pixels, the top two advertisement materials of the priority are displayed; and when the size of the advertisement image generation area is less than 200 pixels * 300 pixels, no advertisement material is displayed because the size of the advertisement image generation area is too small and the user cannot clearly see the displayed advertisement. Based on the size of the advertisement image generation area, the advertisement materials of the corresponding priority are displayed, and the advertisement materials are reorganized and simplified according to the layout to generate an advertisement image.

[0197] In some embodiments, the server delivers the advertisement image to the client corresponding to the user identifier, and the client displays the intelligently generated advertisement image to the user. The server delivers the intelligently generated advertisement image to the client corresponding to the user identifier, and when the user flips to the corresponding page, the client displays the corresponding advertisement image to the user, and the advertisement image can be eliminated in response to an ignore operation, and the detail page of the corresponding advertisement image can be displayed in response to a detail viewing operation.

[0198] The image-based information recommendation method provided in the embodiments of the present application identifies the advertisement materials cut out from the advertisement image based on the AI image recognition technology, defines the priority of each advertisement material, and then automatically combines and intelligently delivers the advertisement materials according to the size (area and shape) of the blank space of the page. When it is detected that the size of the blank area of the page meets the predetermined threshold, the corresponding reorganized advertisement materials are delivered, which not only improves the layout utilization rate, but also effectively improves the exposure rate, consumption rate and conversion rate of the advertisement.

[0199] It can be understood that in the embodiments of the present application, user information and other related data are involved, and when the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0200] The following continues to describe an exemplary structure of the image-based information recommendation apparatus 455-1 provided in the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 4, the image-based information recommendation apparatus 455-1 includes a data acquisition module 410, an information recommendation module 420, a data storage module 430, and a data processing module 440. Figure 2AAs shown, the software modules stored in the image-based information recommendation device 455-1 in the memory 450 can include: a first region module 4551, configured to perform blank area identification processing on an information browsing interface including at least one piece of information to obtain a plurality of blank areas in the information browsing interface; a first material module 4552, configured to perform material identification processing on an image used for information recommendation to obtain a plurality of recommendation materials in the image; a first priority module 4553, configured to obtain a priority of each recommendation material; a first composition module 4554, configured to select the recommendation materials in order from high to low according to the priorities, and compose a material set corresponding to the blank area based on the selected recommendation materials, wherein the size sum of at least one recommendation material included in the material set is adapted to the size of the blank area; and a first display module 4555, configured to display the material set corresponding to the blank area in at least one blank area of the information browsing interface.

[0201] In some embodiments, the first material module 4552 is further configured to: perform preprocessing on the image to obtain a preprocessed image, wherein the preprocessing includes at least one of the following: image denoising processing, image smoothing processing, and image transformation processing; perform feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; perform mapping processing on the image features to obtain probabilities of a plurality of material types corresponding to each sampling pixel block in the image; and perform the following processing for each sampling pixel block: determine a material type with the highest probability as a material type corresponding to the sampling pixel block; and perform merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain a recommendation material corresponding to the same material type.

[0202] In some embodiments, the first material module 4552 is further configured to: perform feature extraction processing on the preprocessed image to obtain three-dimensional features of each pixel block in the preprocessed image; perform spatial sampling processing on a plurality of pixel blocks in the preprocessed image based on spatial features in the three-dimensional features of each pixel block to obtain a plurality of sampling pixel blocks; and splice the three-dimensional features of the plurality of sampling pixel blocks into the image features.

[0203] In some embodiments, the first priority module 4553 is further configured to perform any one of the following processing: obtain a priority preconfigured for each material type, and determine the priority of the recommendation material as the priority of the material type of the recommendation material; obtain a feature parameter of the recommendation material, and determine a priority positively correlated with the feature parameter, wherein the feature parameter includes at least one of the following: an area of the recommendation material, and a degree of interest of the object to be recommended in the recommendation material.

[0204] In some embodiments, the first component module 4554 is further configured to: obtain at least one blank area meeting a blank area condition from the plurality of blank areas as a blank area to be filled; and perform the following processing for each blank area to be filled: query the mapping table for a number of materials corresponding to the size of the blank area; sort the plurality of recommended materials in descending order of priority, and obtain at least one recommended material meeting the number of materials in the front of the sorting; and perform scaling processing on the at least one recommended material based on the size of the blank area, and group the scaled at least one recommended material into a material set.

[0205] In some embodiments, the first component module 4554 is further configured to: obtain at least one blank area meeting a blank area condition from the plurality of blank areas as a blank area to be filled; and perform the following processing for each blank area to be filled: sort the plurality of recommended materials in descending order of priority, and perform the following processing for each recommended material according to the sorting result: when the fillable size of the blank area to be filled is not less than the size of the recommended material, the recommended material is used as a target material for filling the blank area to be filled; and when the fillable size of the blank area to be filled is less than the size of the recommended material, a material set corresponding to the blank area is generated based on the already selected target material.

[0206] In some embodiments, the blank area condition comprises at least one of: the size of the blank area is greater than a size threshold; the distance between the position of the blank area and the center of the information browsing interface is not less than a first distance threshold; and the minimum distance between the blank area and the displayed material set is greater than a second distance threshold.

[0207] In some embodiments, the first material module 4552 is further configured to: perform any one of the following processing for the plurality of candidate images: determine the degree of association of the candidate image with the information content displayed in the information browsing interface, and select the candidate image with the highest degree of association as the image for information recommendation; and determine the degree of interest of the object to be recommended for the candidate image, and select the candidate image with the highest degree of interest as the image.

[0208] In some embodiments, the first area module 4551 is further configured to: obtain a preview interface image in the information browsing interface, the preview interface image comprising at least one information, and the preview interface image being an interface image of the information content displaying the at least one information in the information browsing interface; identify pixels in the preview interface image that do not belong to the information content as background pixels; perform connected processing on the mutually adjacent background pixels of the plurality of background pixels to obtain a plurality of connected areas, and perform cropping processing on each connected area to obtain a plurality of blank areas meeting a preset shape.

[0209] In some embodiments, the first region module 4551 is further configured to: acquire a standard binarization result of the blank region; perform binarization processing on each pixel in the preview interface image to obtain a binarization result of each pixel; perform compensation processing on the binarization result of each pixel to obtain a compensated binarization result of each pixel; and identify a pixel corresponding to the compensated binarization result as a background pixel when the compensated binarization result is the same as the standard binarization result.

[0210] In some embodiments, the first region module 4551 is further configured to: determine an information color value of a representative pixel belonging to information content and a blank color value of a representative pixel not belonging to the information content; when the blank color value is less than the information color value, set a zero value as the standard binarization result of the blank region; and when the blank color value is not less than the information color value, set a non-zero value as the standard binarization result of the blank region.

[0211] The following continues to describe an exemplary structure of the image-based information recommendation apparatus 255 provided by the embodiments of the present application, which is implemented as a software module. As shown in Figure 2B The software module stored in the image-based information recommendation apparatus 255 in the memory 250 can include: a second region module 2551 configured to perform blank region identification processing on an information browsing interface in a terminal to obtain a plurality of blank regions in the information browsing interface; a second material module 2552 configured to perform material identification processing on an image used for information recommendation to obtain a plurality of recommendation materials in the image; a second priority module 2553 configured to acquire a priority of each of the recommendation materials; a second composition module 2554 configured to select the recommendation materials in a descending order of the priorities, and compose the blank regions based on the selected recommendation materials, wherein a sum of sizes of at least one of the recommendation materials included in the material set is adapted to a size of the blank region; and a sending module 2555 configured to send the material set corresponding to the at least one blank region to the terminal.

[0212] The following continues to describe an exemplary structure of the image-based information recommendation apparatus 455-2 provided by the embodiments of the present application, which is implemented as a software module. As shown in Figure 2AAs shown, the software modules stored in the image-based information recommendation device 455-2 in the memory 450 can include a second display module 4556 configured to display at least one piece of information in an information browsing interface; the second display module 4556 is further configured to display a material set in at least one blank area remaining after displaying the at least one piece of information; the material set includes at least one recommended material and is obtained by identifying the image; the sum of the sizes of the at least one recommended material included in the material set is adapted to the size of the blank area; and the recommended materials are sequentially selected according to the priority of the multiple recommended materials in the image from high to low.

[0213] The following continues to describe an exemplary structure of the image-based information recommendation device 455-2 provided by the embodiments of the present application as a software module. As shown in FIG. 4, the software modules stored in the image-based information recommendation device 455-2 in the memory 450 can include a second display module 4556 configured to display at least one piece of information in an information browsing interface; the second display module 4556 is further configured to display a material set in at least one blank area remaining after displaying the at least one piece of information; the material set includes at least one recommended material and is obtained by identifying the image; the sum of the sizes of the at least one recommended material included in the material set is adapted to the size of the blank area; and the recommended materials are sequentially selected according to the priority of the multiple recommended materials in the image from high to low. Figure 2A As shown, the software modules stored in the image-based information recommendation device 455-3 in the memory 450 can include a third display module 4557 configured to display at least part of the content of an article in an article browsing interface; the third display module 4557 is further configured to display a material set in at least one blank area; the at least one blank area is an area in the article browsing interface that is not used to display the part of the content; and the recommended materials in the material set are derived from the image used for information recommendation; and a scaling module 4558 configured to, in response to a scaling operation, display a scaling result of the at least part of the content in the article browsing interface and display a new material set in at least one new blank area; the at least one new blank area is an area in the article browsing interface that is not used to display the scaling result; and the materials in the new material set are derived from the image.

[0214] In some embodiments, the third display module 4557 is further configured to, when the position of the new blank area in the article is the same as the position of the blank area in the article, perform any one of the following processing on the new blank area: synchronously scale the material set displayed in the at least one blank area to obtain a new material set adapted to the at least one new blank area and display the new material set in the at least one new blank area; and adaptively add or remove materials from the material set displayed in the at least one blank area to obtain a new material set adapted to the at least one new blank area and display the new material set in the at least one new blank area.

[0215] The embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the image-based information recommendation method provided by the embodiments of the present application.

[0216] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor executes the image-based information recommendation method provided by the embodiment of the present application, for example, as shown in the following. Figures 3A-3D The image-based information recommendation method is shown.

[0217] In some embodiments, the computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, and the like; and can also be various devices including one or any combination of the above storage.

[0218] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0219] As an example, the executable instructions can but not necessarily correspond to files in a file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code portions).

[0220] As an example, the executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed at multiple sites and interconnected through a communication network.

[0221] In summary, by performing material identification processing on the image for information recommendation in the embodiment of the present application, a plurality of recommendation materials are obtained, the recommendation materials are selected in order from high to low according to the priority of the recommendation materials, the selected recommendation materials are used to form a material set corresponding to a blank area in an information browsing interface, the material set corresponding to the blank area is displayed in the blank area, so that the image can be displayed in the blank area of the information browsing interface, the display resource utilization rate of the information browsing interface and the recommendation flexibility of information recommendation are effectively improved, and since the recommendation materials with higher priority are preferentially displayed, the recommendation efficiency of information recommendation can be improved.

[0222] The above merely provides an example of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An image-based information recommendation method characterized by, The method comprises: Performing blank area identification processing on an information browsing interface comprising at least one piece of information to obtain a plurality of blank areas in the information browsing interface; Performing material identification processing on an image for information recommendation to obtain a plurality of recommendation materials in the image; wherein the material identification processing on the image for information recommendation to obtain a plurality of recommendation materials in the image comprises: performing preprocessing on the image to obtain a preprocessed image, wherein the preprocessing comprises at least one of the following: image denoising processing, image smoothing processing, and image transformation processing; performing feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; performing mapping processing on the image features to obtain probabilities of a plurality of material types corresponding to each sampling pixel block in the image; and performing the following processing for each sampling pixel block: determining a material type with the highest probability as a material type corresponding to the sampling pixel block; and performing merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain a recommendation material corresponding to the same material type; Obtaining a priority of each recommendation material; Selecting the recommendation materials in descending order of the priorities, and composing a material set corresponding to the blank areas based on the selected recommendation materials, wherein the sum of the sizes of at least one recommendation material included in the material set is adapted to the size of the blank area; Displaying the material set corresponding to the blank area in at least one blank area of the information browsing interface.

2. The method of claim 1, wherein, The feature extraction processing on the preprocessed image to obtain image features of the preprocessed image comprises: Performing feature extraction processing on the preprocessed image to obtain three-dimensional features of each pixel block in the preprocessed image; Performing spatial sampling processing on a plurality of pixel blocks in the preprocessed image based on spatial features in the three-dimensional features of each pixel block to obtain a plurality of sampling pixel blocks; Splicing the three-dimensional features of the plurality of sampling pixel blocks into the image features.

3. The method of claim 1, wherein, The obtaining of the priority of each recommendation material comprises: Performing any one of the following processing: Obtaining a priority configured in advance for each material type, and determining the priority of the material type of the recommendation material as the priority of the recommendation material; Obtaining a feature parameter of the recommendation material, and determining a priority positively correlated with the feature parameter, wherein the feature parameter comprises at least one of the following: an area of the recommendation material and a degree of interest of an object to be recommended in the recommendation material.

4. The method of claim 1, wherein, The selecting of the recommendation materials in descending order of the priorities, and the composing of the material set corresponding to the blank areas based on the selected recommendation materials, comprises: Obtaining at least one blank area meeting a blank area condition from the plurality of blank areas, and taking the blank area as a blank area to be filled; Performing the following processing for each blank area to be filled: Querying a number of materials corresponding to the size of the blank area in a mapping table; rank the plurality of recommended materials in descending order of the priority, and obtain at least one recommended material meeting the number of materials in the top of the ranking; perform scaling processing on the at least one recommended material based on the size of the blank area, and group the at least one recommended material after the scaling processing into the material set.

5. The method of claim 1, wherein, The selecting the recommended material in descending order of the priority comprises: obtaining at least one blank area meeting a blank area condition from the plurality of blank areas as a blank area to be filled; performing the following processing for each of the blank area to be filled: ranking the plurality of recommended materials in descending order of the priority, and performing the following processing for each of the recommended material according to the ranking result: when the fillable size of the blank area to be filled is not less than the size of the recommended material, taking the recommended material as a target material for filling the blank area to be filled; when the fillable size of the blank area to be filled is less than the size of the recommended material, generating a material set corresponding to the blank area based on the target material that has been selected.

6. The method according to claim 4 or 5, characterized in that, The blank area condition comprises at least one of the following: the size of the blank area is greater than a size threshold; the distance between the position of the blank area and the center of the information browsing interface is not less than a first distance threshold; the minimum distance between the blank area and the displayed material set is greater than a second distance threshold.

7. The method of claim 1, wherein, The method further comprises: performing any one of the following processing for a plurality of candidate images: determining the relevance of the candidate image to the information content displayed in the information browsing interface, and taking the candidate image with the highest relevance as the image for information recommendation; determining the interest degree of the object to be recommended to the candidate image, and taking the candidate image with the highest interest degree as the image.

8. The method of claim 1, wherein, The blank area identification processing on the information browsing interface comprising at least one information comprises: obtaining a preview interface image corresponding to the information browsing interface, the preview interface image comprising the at least one information; identifying the pixels in the preview interface image that do not belong to information content as background pixels; performing connected processing on the background pixels adjacent to each other to obtain a plurality of connected regions, and performing cropping processing on each of the connected regions to obtain a plurality of blank areas meeting a preset shape.

9. The method of claim 8, wherein, The identifying the pixels in the preview interface image that do not belong to information content as background pixels comprises: obtaining a standard binarization result corresponding to the blank area; performing binarization processing on each pixel in the preview interface image to obtain a binarization result of each pixel; performing compensation processing on the binarization result of each pixel to obtain a compensated binarization result of each pixel; when the compensated binarization result is the same as the standard binarization result, identifying the pixel corresponding to the compensated binarization result as the background pixel.

10. The method of claim 9, wherein, The standard binarization result corresponding to the blank area is obtained, including: determining the information color value of the representative pixel belonging to the information content and the blank color value of the representative pixel not belonging to the information content; when the blank color value is less than the information color value, the zero value is taken as the standard binarization result of the blank area; when the blank color value is not less than the information color value, the non-zero value is taken as the standard binarization result of the blank area.

11. An image-based information recommendation method characterized by comprising: The method comprises: blank area identification processing is performed on an information browsing interface including at least one information in a terminal to obtain a plurality of blank areas in the information browsing interface; material identification processing is performed on an image for information recommendation to obtain a plurality of recommendation materials in the image; wherein the material identification processing performed on the image for information recommendation to obtain a plurality of recommendation materials in the image comprises: pre-processing the image to obtain a pre-processed image, wherein the pre-processing comprises at least one of the following: image denoising processing, image smoothing processing, image transformation processing; performing feature extraction processing on the pre-processed image to obtain image features of the pre-processed image; performing mapping processing on the image features to obtain the probability of each sampling pixel block in the image corresponding to a plurality of material types; for each sampling pixel block, the following processing is performed: determining the material type with the highest probability as the material type corresponding to the sampling pixel block; merging adjacent sampling pixel blocks corresponding to the same material type to obtain recommendation materials corresponding to the same material type; obtaining the priority of each recommendation material; selecting the recommendation materials in order from high to low according to the priority, and composing a material set corresponding to the blank area based on the selected recommendation materials, wherein the size of at least one recommendation material included in the material set is adapted to the size of the blank area; sending at least one material set corresponding to the blank area to the terminal.

12. An image-based information recommendation method characterized by comprising: The method comprises: displaying at least one information in an information browsing interface; displaying a material set in at least one blank area remaining after displaying the at least one information; The material set includes at least one recommended material, the recommended material is obtained by identifying an image used for information recommendation, the sum of the sizes of the at least one recommended material included in the material set is adapted to the size of the blank area, the recommended material is obtained in the order from high to low of the priority of a plurality of recommended materials in the image; wherein the recommended material is obtained by the following steps: preprocessing the image to obtain a preprocessed image, wherein the preprocessing includes at least one of the following: image denoising processing, image smoothing processing, image transformation processing; performing feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; performing mapping processing on the image features to obtain the probability of a plurality of material types corresponding to each sampling pixel block in the image; for each sampling pixel block, the following processing is performed: determining the material type with the highest probability as the material type corresponding to the sampling pixel block; and performing merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain recommended materials corresponding to the same material type.

13. An image-based information recommendation method characterized by comprising: The method comprises: displaying at least part of the content of an article in an article browsing interface; displaying a material set in at least one blank area, wherein the at least one blank area is an area in the article browsing interface that is not used to display the part of the content, and the recommended materials in the material set are derived from an image used for information recommendation, wherein the recommended materials are obtained by the following steps: preprocessing the image to obtain a preprocessed image, wherein the preprocessing includes at least one of the following: image denoising processing, image smoothing processing, image transformation processing; performing feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; performing mapping processing on the image features to obtain the probability of a plurality of material types corresponding to each sampling pixel block in the image; for each sampling pixel block, the following processing is performed: determining the material type with the highest probability as the material type corresponding to the sampling pixel block; and performing merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain recommended materials corresponding to the same material type; in response to a zoom operation, displaying the zoom result of the at least part of the content in the article browsing interface, and displaying a new material set in at least one new blank area; wherein the at least one new blank area is an area in the article browsing interface that is not used to display the zoom result, and the materials of the new material set are derived from the image.

14. The method of claim 13, wherein, The displaying of the new material set in the at least one new blank area comprises: when the position of the new blank area in the article is the same as the position of the blank area in the article, any one of the following processing is performed for the new blank area: performing synchronous zoom processing on the material set displayed in the at least one blank area to obtain a new material set adapted to the at least one new blank area, and displaying the new material set in the at least one new blank area; Adaptively performing material adding and reducing processing on the material set displayed in the at least one blank area to obtain a new material set adapted to the at least one new blank area, and displaying the new material set in the at least one new blank area.

15. An image-based information recommendation apparatus characterized by comprising: The device comprises: A first area module configured to perform blank area identification processing on an information browsing interface comprising at least one information to obtain a plurality of blank areas in the information browsing interface; A first material module configured to perform material identification processing on an image used for information recommendation to obtain a plurality of recommendation materials in the image; wherein the first material module is further configured to: perform preprocessing on the image to obtain a preprocessed image, wherein the preprocessing comprises at least one of the following: image denoising processing, image smoothing processing, and image transformation processing; perform feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; perform mapping processing on the image features to obtain probabilities of a plurality of material types corresponding to each sampling pixel block in the image; and perform the following processing for each sampling pixel block: determine a material type with the highest probability as a material type corresponding to the sampling pixel block; and perform merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain a recommendation material corresponding to the same material type; A first priority module configured to obtain a priority of each recommendation material; A first composition module configured to select the recommendation materials in a descending order of the priorities, and compose a material set corresponding to the blank area based on the selected recommendation materials, wherein the sum of the sizes of at least one recommendation material included in the material set is adapted to the size of the blank area; A first display module configured to display the material set corresponding to the blank area in at least one blank area of the information browsing interface.

16. The apparatus of claim 15, wherein, The first material module is further configured to: perform feature extraction processing on the preprocessed image to obtain three-dimensional features of each pixel block in the preprocessed image; perform spatial sampling processing on a plurality of pixel blocks in the preprocessed image based on spatial features in the three-dimensional features of each pixel block to obtain a plurality of sampling pixel blocks; splice the three-dimensional features of the plurality of sampling pixel blocks into the image features.

17. The apparatus of claim 15, wherein, The first composition module is further configured to: obtain at least one blank area meeting a blank area condition from the plurality of blank areas as a blank area to be filled; perform the following processing for each blank area to be filled: sort the plurality of recommendation materials in a descending order of the priorities, and perform the following processing for each recommendation material according to the sorting result: when the fillable size of the blank area to be filled is not smaller than the size of the recommendation material, take the recommendation material as a target material used for filling the blank area to be filled; when the fillable size of the blank area to be filled is smaller than the size of the recommendation material, generate a material set corresponding to the blank area based on the already selected target material.

18. The apparatus of claim 17, wherein, The blank area condition comprises at least one of the following: a size of the blank area is greater than a size threshold; a distance between a position of the blank area and a center of the information browsing interface is not less than a first distance threshold; and a minimum distance between the blank area and a displayed material set is greater than a second distance threshold.

19. An image-based information recommendation apparatus characterized by comprising: The apparatus comprises: A second area module configured to perform blank area identification processing on an information browsing interface comprising at least one piece of information in a terminal to obtain a plurality of blank areas in the information browsing interface; A second material module configured to perform material identification processing on an image used for information recommendation to obtain a plurality of recommended materials in the image; wherein the second material module is further configured to: perform preprocessing on the image to obtain a preprocessed image, wherein the preprocessing comprises at least one of the following: image denoising processing, image smoothing processing, and image transformation processing; perform feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; perform mapping processing on the image features to obtain probabilities of a plurality of material types corresponding to each sampling pixel block in the image; and perform the following processing for each sampling pixel block: determine a material type with the highest probability as a material type corresponding to the sampling pixel block; and perform merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain a recommended material corresponding to the same material type; A second priority module configured to obtain a priority of each recommended material; A second composition module configured to select the recommended materials in a descending order of the priorities, and to compose a material set corresponding to the blank area based on the selected recommended materials, wherein a sum of sizes of at least one recommended material included in the material set is adapted to a size of the blank area; A sending module configured to send the material set corresponding to the at least one blank area to the terminal.

20. An image-based information recommendation apparatus characterized by comprising: The apparatus comprises: A second display module configured to display at least one piece of information in an information browsing interface; The second display module is further configured to display a material set in at least one blank area remaining after the at least one information is displayed; the material set comprises at least one recommended material and is obtained by identifying the image; the sum of the sizes of the at least one recommended material comprised in the material set is adapted to the size of the blank area; the recommended materials are sequentially selected according to the priority of the multiple recommended materials in the image from high to low; the recommended materials are obtained by the following steps: preprocessing the image to obtain a preprocessed image, wherein the preprocessing comprises at least one of image denoising processing, image smoothing processing and image transformation processing; performing feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; performing mapping processing on the image features to obtain the probability of multiple material types corresponding to each sampling pixel block in the image; and performing the following processing on each sampling pixel block: determining the material type with the highest probability as the material type corresponding to the sampling pixel block; and performing merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain recommended materials corresponding to the same material type.

21. An image-based information recommendation apparatus characterized by comprising: The apparatus comprises: A third display module configured to display at least part of the content of an article in an article browsing interface; The third display module is further configured to display a material set in at least one blank area, wherein the at least one blank area is an area in the article browsing interface that is not used to display the part of the content, and the recommended materials in the material set are derived from an image used for information recommendation; wherein the recommended materials are obtained by the following steps: preprocessing the image to obtain a preprocessed image, wherein the preprocessing comprises at least one of image denoising processing, image smoothing processing and image transformation processing; performing feature extraction processing on the preprocessed image to obtain image features of the preprocessed image; performing mapping processing on the image features to obtain the probability of multiple material types corresponding to each sampling pixel block in the image; and performing the following processing on each sampling pixel block: determining the material type with the highest probability as the material type corresponding to the sampling pixel block; and performing merging processing on adjacent sampling pixel blocks corresponding to the same material type to obtain recommended materials corresponding to the same material type; A scaling module configured to, in response to a scaling operation, display the scaling result of the at least part of the content in the article browsing interface and display a new material set in at least one new blank area; wherein the at least one new blank area is an area in the article browsing interface that is not used to display the scaling result, and the materials in the new material set are derived from the image.

22. An electronic device, characterized in that: The electronic device comprises: A memory configured to store executable instructions; A processor configured to execute the executable instructions stored in the memory to implement the image-based information recommendation method of any one of claims 1 to 10, 11, 12 and 13-14.

23. A computer-readable storage medium storing executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising: The executable instructions, when executed by a processor, implement the image-based information recommendation method according to any one of claims 1-10, 11, 12, 13-14.

24. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the image-based information recommendation method according to any one of claims 1-10, 11, 12, 13-14.

Images