Server, display equipment and dynamic image cutting method

Abstract

The embodiment of the invention provides a server, a display device and a dynamic image cutting method, and the method can receive an image acquisition request sent by the display device, and the image acquisition request comprises a type parameter and a data source identifier. And in response to an image acquisition request, acquiring a first image according to the type parameter and the data source identifier, and generating a cutting parameter according to the data source identifier and the original size of the first image. Wherein the cutting parameters comprise a target size and a target aspect ratio. And identifying a key region of the first image according to the cutting parameter and a region extraction algorithm, and cutting the first image according to the key region to generate a second image. And sending the second image to the display device, so that the display device displays the second image in the preset display area. According to the method, the image adaptive to the display equipment can be dynamically generated, the image display effect of the display equipment can be improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a server, a display device, and a method for dynamically cropping an image. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. Taking smart TV as an example, smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users.

[0003] The display device can provide different types of media through different access methods and media playback applications, such as live TV programs, on-demand TV programs, etc.; and the display device can also expand more functions through other types of applications, such as conference applications, whiteboard applications, etc. Among them, the user interface is the key link for the user to interact with the display device, such as the home page interface, etc.; the user interface may include multiple image views, and the image view may be a dynamic image (such as a preview dynamic image of the program, etc.) or a static image (such as a static preview image of the program or an application icon, etc.). The user can input corresponding control instructions to the display device through the user interface to enable the display device to implement corresponding functions, such as playing media resources, switching application interfaces, etc.

[0004] However, each image view in the user interface displays the original image by cropping it at a fixed ratio and size. Due to the increase in the types of applications, the above image cropping method may cause the cropped image to be too large, too small, or have lost details, thereby affecting the display effect of the image in the display device and reducing the user experience. Summary of the invention

[0005] The present application provides a server, a display device and a method for dynamically cropping an image to solve the problem of poor image display effect in the display device.

[0006] In a first aspect, some embodiments of the present application provide a server, comprising a communication module and a control module. The communication module is configured to establish communication with a display device, and the control module is configured to execute the following program steps:

[0007] receiving an image acquisition request sent by the display device, wherein the image acquisition request includes a type parameter and a data source identifier;

[0008] In response to the image acquisition request, acquiring a first image according to the type parameter and the data source identifier;

[0009] generating cropping parameters according to the data source identifier and the original size of the first image, the cropping parameters including a target size and a target aspect ratio;

[0010] identifying a key area of ​​the first image according to the cropping parameters and the area extraction algorithm;

[0011] Cropping the first image according to the key area to generate a second image; the image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio;

[0012] The second image is sent to the display device, so that the display device displays the second image in a preset display area.

[0013] The above technical solution has the following advantages or beneficial effects: dynamically generating corresponding cropping parameters through the data source identifier and the original size of the first image, and extracting key areas through the area extraction algorithm to ensure that there is no information loss in the cropped second image, so that the server can send a suitable second image to the display device, thereby improving the display effect of the image in the display device.

[0014] In some embodiments of the present application, the control module executes obtaining the first image according to the type parameter and the data source identifier, and is specifically configured to: obtain reference data according to the type parameter and the data source identifier, the reference data including media data and historical data, the media data including media ID, media tag, media title and media description information, and the historical data including at least one of viewing history, click record and rating record; extract user tags from the historical data; convert the media data into a target matrix, and convert the user tags into a target vector, the target matrix is ​​a matrix generated based on word frequency and inverse document frequency, the target matrix includes a baseline vector of media tags based on word frequency and inverse document frequency, and the target vector is a vector generated based on word frequency and inverse document frequency; calculate cosine similarity according to the target matrix and the target vector, the cosine similarity is the similarity between the target vector and the baseline vector; obtain the target media data with the highest cosine similarity; output the image corresponding to the target media data through a recommendation function to obtain the first image.

[0015] The above technical solution has the following advantages or beneficial effects: media images are recommended through type parameters and data source identifiers, and the priority of media image recommendation is determined based on multiple dimensions and by analyzing historical data, so as to achieve more accurate image recommendation, so that the server can obtain a more suitable first image.

[0016] In some embodiments of the present application, after the control module executes the step of acquiring reference data according to the type parameter and the data source identifier, the control module is also configured to: load a pre-trained image recognition model, wherein the image recognition model includes a convolutional neural network; input the media data into the image recognition model so that the image recognition model outputs a classification prediction result of the media data; if the classification prediction result is in a preset screening classification list, delete the media data; if the classification prediction result is not in the screening classification list, retain the media data.

[0017] The above technical solution has the following advantages or beneficial effects: through an image recognition model including a convolutional neural network, image recognition is performed on images corresponding to the acquired media data based on a deep learning algorithm, and images that do not meet the standards are screened out according to a preset screening classification list, so that the server can send images to the display device more accurately.

[0018] In some embodiments of the present application, the control module generates cropping parameters according to the data source identifier and the original size of the first image, and is specifically configured as follows: reading the original size of the first image according to the uniform resource locator of the first image; calculating the original aspect ratio according to the original size; reading the target aspect ratio corresponding to the data source identifier; calculating the target size of the second image according to the target aspect ratio and the original aspect ratio; and setting the cropping parameters according to the target aspect ratio and the target size.

[0019] The above technical solution has the following advantages or beneficial effects: the corresponding cropping size and aspect ratio are calculated through the data source identifier and the original size of the first image, so that the image cropped according to the cropping size and aspect ratio presents a more appropriate display effect.

[0020] In some embodiments of the present application, the control module executes identification of the key area of ​​the first image based on the cropping parameters and the region extraction algorithm, and is specifically configured to: detect the key features of the first image based on the region extraction algorithm; obtain the main area of ​​the key features in the first image; calculate the key area according to the cropping parameters and the main area, the key area includes the main area, and the size of the key area is equal to the target size.

[0021] The above technical solution has the following advantages or beneficial effects: key features in the first image are identified through an automated adaptive region extraction algorithm, and the key features are retained when the first image is cropped, so that the cropped image can accurately retain the original information of the first image.

[0022] In some embodiments of the present application, the control module is further configured to: connect to a remote dictionary service; create a cache function based on the remote dictionary service; cache media data through the cache function to generate a cache key for the media data, wherein the cache key is associated with an image identifier of an image corresponding to the media data; traverse the usage frequency of the cache key; if the usage frequency is less than or equal to a frequency threshold, delete the cache key; if the usage frequency is greater than the frequency threshold, retain the cache key to generate local cache data.

[0023] The above technical solution has the following advantages or beneficial effects: setting a pre-caching mechanism to cache images corresponding to media data locally, and deleting images with less frequent use during caching, retaining popular images with higher frequency of use, and improving the prediction accuracy of the pre-caching mechanism.

[0024] In some embodiments of the present application, the control module executes obtaining a first image according to the type parameter and the data source identifier, and is specifically configured to: obtain a target identifier according to the type parameter and the data source identifier; query the image identifier of the local cached data; if the image identifier of the local cached data includes the target identifier, extracting the image corresponding to the target identifier in the local cached data to obtain the first image.

[0025] The above technical solution has the following advantages or beneficial effects: when obtaining the first image in response to an image acquisition request, it is queried whether the local cache data includes the image to be obtained. If it is included, the image is extracted from the local cache data, which can speed up the acquisition of the first image and improve the response efficiency of the server.

[0026] In some embodiments of the present application, the control module executes caching of media data through the caching function, and is specifically configured to: detect a focus area of ​​an image corresponding to the media data, wherein the focus area is determined based on a position of a control focus in a historical operation log; perform block processing on the image corresponding to the media data through the focus area to generate at least two block data; and cache the block data to the remote dictionary service.

[0027] The above technical solution has the following advantages or beneficial effects: when pre-caching images corresponding to media data, the image area that the user uses more frequently in the image is determined according to the focus position of the user's historical operations, and is used as the focus area; the cached single image is divided into blocks according to the focus area and the non-focus area, which can be used to improve the speed of extracting a single image in the local cache data.

[0028] In a second aspect, some embodiments of the present application also provide a display device, including a display, a communication device, and a controller. The display is configured to display a user interface, the communication device is configured to establish communication with a server, and the controller is configured to execute the following program steps:

[0029] In response to an interface display instruction input by a user, detecting a type parameter and a data source identifier associated with the interface display instruction;

[0030] Generate an image acquisition request according to the type parameter and the data source identifier;

[0031] Sending the image acquisition request to the server;

[0032] receiving a second image sent by the server, the second image being an image sent by the server in response to the image acquisition request, the second image being an image generated by cropping the first image according to the key area, the first image being an image acquired according to the type parameter and the data source identifier, the key area being identified in the first image according to cropping parameters and an area extraction algorithm, the cropping parameters being parameters generated according to the data source identifier and an original size of the first image, the cropping parameters including a target size and a target aspect ratio, the image size of the second image being the target size, and the image aspect ratio of the second image being the target aspect ratio;

[0033] The display is controlled to display the second image in a preset display area of ​​the user interface.

[0034] The above technical solution has the following advantages or beneficial effects: according to the interface display instruction sent by the user on the display device, an image acquisition request including a data source identifier and a type parameter can be generated, and the image acquisition request can be sent to the server, so that the server can dynamically generate corresponding cropping parameters through the data source identifier and the original size of the first image, and extract key areas through the area extraction algorithm to ensure that the cropped second image will not lose information. In this way, the second image displayed by the server to the display device can be more adapted to the current display effect of the display device.

[0035] In a third aspect, some embodiments of the present application further provide a method for dynamic cropping of an image, comprising the following steps:

[0036] Receiving an image acquisition request sent by a display device, wherein the image acquisition request includes a type parameter and a data source identifier;

[0037] In response to the image acquisition request, acquiring a first image according to the type parameter and the data source identifier;

[0038] generating cropping parameters according to the data source identifier and the original size of the first image, the cropping parameters including a target size and a target aspect ratio;

[0039] identifying a key area of ​​the first image according to the cropping parameters and the area extraction algorithm;

[0040] Cropping the first image according to the key area to generate a second image; the image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio;

[0041] The second image is sent to the display device, so that the display device displays the second image in a preset display area.

[0042] The above technical solution has the following advantages or beneficial effects: the corresponding cropping parameters are dynamically generated through the data source identifier and the original size of the first image, and the key area is extracted through the area extraction algorithm to ensure that there is no information loss in the cropped second image, so that a suitable second image can be sent to the display device, thereby improving the display effect of each image in the display device.

[0043] From the above technical aspects, it can be known that some embodiments of the present application provide a server, a display device, and a dynamic cropping method for an image, the method can receive an image acquisition request sent by a display device, and the image acquisition request includes a type parameter and a data source identifier. Then, in response to the image acquisition request, a first image is acquired according to the type parameter and the data source identifier, and cropping parameters are generated according to the data source identifier and the original size of the first image. Among them, the cropping parameters include a target size and a target aspect ratio. According to the cropping parameters and the area extraction algorithm, the key area of ​​the first image is identified, and then the first image is cropped according to the key area to generate a second image. The second image is sent to the display device so that the display device displays the second image in a preset display area. The method can dynamically generate an image adapted to the display device, which can improve the image display effect of the display device and enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A schematic diagram of a usage scenario of a server provided in some embodiments of the present application;

[0046] Figure 2A schematic diagram of the hardware configuration of a server provided in some embodiments of the present application;

[0047] Figure 3 A schematic diagram of software configuration of a server provided in some embodiments of the present application;

[0048] Figure 4 A schematic diagram of the display effect of the homepage application provided in some embodiments of the present application;

[0049] Figure 5 A schematic diagram of interaction between a display device and a server provided in some embodiments of the present application;

[0050] Figure 6 A schematic diagram of a flow chart of a method for dynamic image cropping provided in some embodiments of the present application;

[0051] Figure 7 An interactive diagram of a method for dynamic image cropping provided in some embodiments of the present application;

[0052] Figure 8 A schematic diagram of a process of acquiring a first image provided in some embodiments of the present application;

[0053] Fig. 9 A schematic diagram of a process for screening images based on image recognition technology provided in some embodiments of the present application;

[0054] Fig.10 A schematic diagram of a process for identifying key areas provided in some embodiments of the present application;

[0055] Fig.11 A schematic diagram of a process flow of a pre-caching mechanism provided in some embodiments of the present application;

[0056] Fig.12 A schematic diagram of the process of block caching provided in some embodiments of the present application;

[0057] Fig.13 An architectural diagram of a display device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0058] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0059] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0060] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0061] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0062] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0063] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0064] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0065] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0066] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0067] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0068] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0069] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0070] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0071] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0072] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0073] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0074] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0075] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0076] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0077] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0078] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0079] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0080] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0081] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0082] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, Figure 3 A software configuration diagram of a display device provided for some embodiments of the present application. In some embodiments, the system of the display device 200 can be divided into three layers, namely, an application layer, a middleware layer, and a hardware layer from top to bottom.

[0083] The application layer mainly includes applications on the TV and the application framework. Among them, applications are mainly applications developed based on the browser, such as HTML5 APPs and native applications.

[0084] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0085] Native apps can support online or offline, message push or local resource access.

[0086] The middleware layer includes various TV protocols, multimedia protocols, system components and other middleware. The middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purpose of resource sharing and function sharing.

[0087] The hardware layer mainly includes HAL interface, hardware and driver. Among them, HAL interface is the unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0088] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0089] In some embodiments, the application layer of the display device 200 may be configured with different types of applications, such as media playback applications, conference applications, multimedia applications, reading applications, or game applications, etc. In order to facilitate users to use the above applications, the display device 200 may display icons or preview images of the above applications through application interfaces such as the home page interface to assist users in interacting with the display device 200.

[0090] In some embodiments, the user interface may include multiple views, such as an image view and a text view. The image view may be used to present dynamic images or static images, which is not limited in this application. The image view may be used to display application icons of applications or preview images and poster images of media asset items.

[0091] For example, Figure 4 In the schematic diagram of the homepage application effect, the display device 200 can display different images in different areas, including an application icon 401 for starting an application and a media asset poster image 402. The poster image 402 can be a dynamic image composed of multiple frames of images, which can be used to introduce the summary content of the current media asset.

[0092] For ease of description, the embodiments of the present application uniformly represent the images displayed on the user interface, such as the preview images, poster images, and application icons, as media asset images.

[0093] In some embodiments, the display device 200 may receive an interface display instruction input by a user, and send a request to the server 400 in response to the interface display instruction, so that the server 400 returns the corresponding media asset image to the display device 200 according to the request. Figure 5 As shown, the user can trigger the display device 200 to display a home page interface, which includes multiple media asset items. In order to determine the media asset items included in the home page interface, the display device 200 can send a corresponding acquisition request to the server 400.

[0094] In order to improve the display effect of media images, in some embodiments, after the server 400 receives the acquisition request sent by the display device 200, it can query the corresponding media image according to the acquisition request, and then crop the media image according to a preset fixed ratio and size, and then send the cropped image to the display device 200 for display.

[0095] However, due to the increase in the types of applications, the above-mentioned cropping method with fixed ratios and sizes cannot adapt well to the usage scenarios of the display device 200. For example, the cropped image may be too large or too small; or invalid information in the cropped image may occupy a large space, resulting in a waste of display space; or, the cropped image may have information loss problems, resulting in poor display effect of the media image in the display device 200, thereby reducing the user experience.

[0096] To this end, some embodiments of the present application provide a server 400, which can dynamically generate cropping parameters suitable for the display device 200, and adaptively crop the image in combination with a region extraction algorithm, so that the cropped image can retain more effective information, thereby enhancing the image display effect of the display device 200.

[0097] like Figure 6-Figure 7 As shown, in some embodiments, the server 400 may include a communication module 410 and a control module 420. The communication module 410 is configured to establish a communication connection with the display device 200, and the control module 420 is configured to execute the following program steps:

[0098] S601: Receive an image acquisition request sent by the display device, where the image acquisition request includes a type parameter and a data source identifier.

[0099] The server 400 may receive the image acquisition request sent by the display device 200 through the communication module 410. The image acquisition request may include a type parameter and a data source identifier. The type parameter may be used to characterize the type of image currently displayed by the display device 200 and the type of interface corresponding to the image; the data source identifier may be used to characterize the identifier of the partner corresponding to the image.

[0100] In some embodiments, the display device 200 may detect the type parameter and data source identifier associated with the interface display instruction in response to the interface display instruction input by the user, and then generate an image acquisition request according to the detected type parameter and data source identifier, and send the image acquisition request to the server 400. There may be multiple data source identifiers corresponding to a single interface, such as a home page interface may include media asset images of multiple partners.

[0101] For example, after the user starts the display device 200, the display device 200 can automatically display the home page interface. At this time, the power-on instruction includes the interface display instruction of the home page interface. In response to the instruction, the display device 200 can detect the interface type associated with the home page interface and the data source identifier corresponding to each media image in the home page interface. Then, an image acquisition request is generated according to the interface type and data source identifier of the home page interface, and then the image acquisition request is sent to the display device 200.

[0102] S602: In response to the image acquisition request, acquire a first image according to the type parameter and the data source identifier.

[0103] After receiving the image acquisition request sent by the display device 200, the server 400 can respond to the image acquisition request and acquire the first image according to the type parameter and the data source identifier. The type parameter may correspond to URLs (Uniform Resource Locator) of multiple images, and the server 400 may query and acquire the media image according to the URL and the data source identifier to obtain the first image; or the type parameter may correspond to a media image of a specified category, and the server 400 may acquire the media image according to the specified category and the data source identifier.

[0104] like Figure 8 As shown, in some embodiments, when the server 400 acquires the first image, reference data can be acquired according to the type parameter and the data source identifier. The reference data may include media data and historical data, the media data includes media ID, media tag, media title and media description information, and the historical data includes at least one of viewing history, click record and rating record. Then, the server 400 can extract user tags from the historical data, convert the media data into a target matrix, and convert the user tags into a target vector. The target matrix is ​​a matrix generated based on term frequency and inverse document frequency, the target matrix includes a reference vector of the media tag based on term frequency and inverse document frequency, and the target vector is a vector generated based on term frequency and inverse document frequency, that is, the target matrix is ​​TF-IDF (Term Frequency-Inverse Document Frequency), and the target vector is a TF-IDF vector.

[0105] After the server 400 converts the target matrix and the target vector, it calculates the cosine similarity based on the target matrix and the target vector, and obtains the target media data with the highest cosine similarity. The cosine similarity is the similarity between the target vector and the reference vector. Then, the image corresponding to the target media data is output (i.e., printed) through the recommendation function to obtain the first image

[0106] That is to say, server 400 can recommend media images based on a multi-dimensional collaborative recommendation algorithm, combined with data such as media columns, historical media, user tags, and click rankings; at the same time, it can use big data technology to analyze the content of different media columns, score and sort based on user viewing time, satisfaction, and traffic, and dynamically adjust the recommendation priority of media images.

[0107] For example, server 400 can use the read_csv function in the pandas library to load media data from a file named media_data.csv into a DataFrame named media_data. This DataFrame contains information about the media data, one of which is named tags, which is used to store tags associated with each media data. Then, define the recommend_images function (i.e., the recommendation function) to receive user_id and optional num_recommendations to return media images for a given user ID. Inside the function, first call the get_user_history(user_id) function, which returns the user's historical data based on the user ID. Then, use the extract_tags_from_history(user_history) function to extract a set of tags from the historical data to represent the user's interests and preferences. Use the TfidfVectorizer class to perform TF-IDF vectorization on the tags column in the media_data DataFrame. Server 400 then uses the same TfidfVectorizer instance to convert user tags into TF-IDF vectors. Then use the cosine_similarity function to calculate the cosine similarity between the user's TF-IDF vector and the TF-IDF matrix of all media data. Sort the calculated cosine similarities and select the index of the media image corresponding to the first num_recommendations media data with the highest similarity. Using this index, select the corresponding row from the media_data DataFrame to form a recommendation list. Finally, call the recommend_images function, pass in a user ID, and print the returned recommendation list.

[0108] like Fig. 9 As shown, in some embodiments, in order to improve the accuracy of the first image acquisition, after the server 400 acquires the reference data, it can also load a pre-trained image recognition model (S901). The image recognition model includes a convolutional neural network. Then the media asset data is input into the image recognition model (S902) so that the image recognition model outputs the classification prediction result of the media asset data. If the classification prediction result is in the preset screening classification list, the media asset data is deleted (S903); and if the classification prediction result is not in the screening classification list, the media asset data is retained (S904).

[0109] That is to say, in the process of acquiring the first image, the server 400 can filter out media images of a specific category through the image recognition model to ensure that the retained media images can meet the display requirements.

[0110] In some embodiments, the image recognition model can be initially recognized by a pre-trained ResNet (Residual Network) or Inception (Inception Network) model, and then further optimized by a self-training model to improve the recognition accuracy of the classification prediction results.

[0111] In some embodiments, the image recognition model may also be configured with a real-time feedback mechanism, and the server 400 may continuously update and optimize the image recognition model based on user feedback on the display device 200, thereby improving the accuracy and robustness of the image recognition model.

[0112] For example, server 400 can import the required TensorFlow and NumPy libraries, as well as the ResNet50 model and related image preprocessing functions in the TensorFlow Keras application. ResNet50 is then used to load a ResNet50 model pre-trained on the ImageNet dataset. This model is an image recognition model that can recognize objects and scenes in the ImageNet dataset. The function receives an image path image_path as input, and uses the image.load_img function to load the image, and specifies the target size as the target pixel. The loaded image is then converted to a NumPy array, and a dimension is added to meet the requirements of the model input. Then, the preprocess_input function is used to preprocess the image, including normalization, etc., to meet the input requirements of the ResNet50 model. The preprocessed image is predicted using the pretrained ResNet50 model to obtain the classification prediction result preds. The classification prediction result is decoded into a readable category name and corresponding probability by the decode_predictions function. Accordingly, the server 400 can traverse the decoded prediction results and check whether the label of each predicted category is in the list of excluded categories. If any category in the list of excluded categories is found, the function returns True, indicating that an image that does not conform is detected; otherwise, False is returned, indicating that the image is available.

[0113] In some embodiments, the screening category lists corresponding to different user tags may also be different. For example, the user ID may be set with a corresponding age group, and the server 400 may screen out media asset images that do not meet the age group according to the age group corresponding to the user ID.

[0114] S603: Generate cropping parameters according to the data source identifier and the original size of the first image, where the cropping parameters include a target size and a target aspect ratio.

[0115] After acquiring the first image, the server 400 may generate corresponding cropping parameters according to the data source identifier in the image acquisition request and the original size of the first image, wherein the cropping parameters include the target size and target aspect ratio of the image to be cropped.

[0116] In some embodiments, the server 400 reads the original size of the first image according to the URL of the first image, and calculates the original aspect ratio according to the original size. Then, the server 400 reads the target aspect ratio corresponding to the data source identifier, and calculates the target size of the second image according to the target aspect ratio and the original aspect ratio. Then, the cropping parameters are set according to the target aspect ratio and the target size.

[0117] Among them, the aspect ratios of the media images corresponding to different data source identifiers are different, such as the aspect ratio corresponding to data source A is 16:9, and the aspect ratio corresponding to data source B is 4:3. The server 400 can determine the current data source according to the data source identifier, and then calculate the appropriate target size and target aspect ratio in combination with the original size of the first image and the aspect ratio corresponding to the data source. For example, the aspect ratio corresponding to the original size of the first image is 16:9, and the image aspect ratio of the data source is 4:3. The server 400 can calculate the corresponding target size according to 4:3 and the original size, and the target aspect ratio corresponding to the target size is 4:3.

[0118] It can be understood that the specific size and calculation method in the above example are only an exemplary description. The method of calculating the cropping parameters in the embodiment of the present application can also be combined with other optimization algorithms, which is not limited in this application.

[0119] S604: Identify key areas of the first image according to the cropping parameters and the area extraction algorithm.

[0120] After calculating the cropping parameters, the server 400 can also identify key areas of the first image, such as subject features such as tasks and roles in the image, based on the cropping parameters and the region extraction algorithm. In this way, the server 400 can crop the first image based on the identified key areas, thereby avoiding problems such as information loss in the first image.

[0121] like Fig.10 As shown, in some embodiments, the server 400 can detect the key features of the first image based on the region extraction algorithm (S1001). Then obtain the key features in the main area of ​​the first image (S1002), and calculate the key area according to the cropping parameters and the main area (S1003). Among them, the key area includes the main area, and the size of the key area is equal to the target size.

[0122] That is, the server 400 can identify the subject in the first image through the region extraction algorithm, and the image region of the subject in the first image is the subject region. The server 400 can calculate the key region corresponding to the image by combining the subject region and the cropping parameters, so that the key region can contain the subject of the first image and can be equal to the above target size.

[0123] S605: Crop the first image according to the key area to generate a second image; the image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio.

[0124] After identifying the above-mentioned key area, the server 400 crops the first image according to the identified key area to generate a second image, that is, the second image is an image of the first image cropped according to the key area, the image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio.

[0125] For example, the server 400 can import the Image module of Python ImagingLibrary through the from PIL import Image statement; then, define the get_image_dimensions function, which accepts the URL of an image as a parameter. Then, define the adaptive_crop function, which accepts three parameters: image_path (image path), target_width (width of the target size) and target_height (height of the target size). Inside the function, the original size of the image, that is, the original width and the original height, is obtained through img.size. Then, the left boundary left, the upper boundary top, the right boundary right and the lower boundary bottom of the cropping area are calculated. Then, the key area of ​​the content of the first image is identified, the center of the cropping area is determined by the coordinate center of the key area, and then the image is cropped by img.crop (left, top, right, bottom) in combination with the calculated cropping area. Finally, the cropped_image.sav is used to save the second image generated after cropping to the cropped_image.jpg file in the current directory to achieve adaptive cropping of the image.

[0126] S606: Send the second image to the display device, so that the display device displays the second image in a preset display area.

[0127] After the server 400 crops the first image into the second image, it can send the second image to the display device 200 for display. In this way, the image received by the display device 200 is the image whose size is dynamically set by the server 400 according to the request, and the key features are retained in the image, so that the image displayed by the display device 200 is more in line with the user's expectations, thereby improving the user's viewing experience.

[0128] like Fig.11 As shown, in some embodiments, in order to improve the response speed of the server 400, the server 400 can also connect to a remote dictionary service (Redis). Then, a cache function is created based on the remote dictionary service, and the media data is cached through the cache function to generate a cache key for the media data. Among them, the cache key is associated with the image identifier of the image corresponding to the media data. Then traverse the usage frequency of the cache key. If the usage frequency is less than or equal to the frequency threshold, the cache key is deleted; if the usage frequency is greater than the frequency threshold, the cache key is retained to generate local cache data.

[0129] That is to say, the server 400 can pre-cache the hottest images locally to optimize the query efficiency of the images. In the process of caching the images, the server 400 can also delete the images with less frequent use according to the frequency of use of the images, thereby reducing the occupancy of unnecessary cache.

[0130] Accordingly, in some embodiments, when acquiring the first image, the server 400 may obtain the target identifier according to the type parameter and the data source identifier, and then query the image identifier of the local cache data. If the image identifier of the local cache data includes the target identifier, the image corresponding to the target identifier is extracted from the local cache data to obtain the first image. Otherwise, the server 400 may obtain the first image according to the type parameter and the data source identifier. Figure 8 The method shown acquires the first image from the corresponding data source.

[0131] For example, server 400 can initialize a Redis connection. Then receive two parameters through the cache_image(image_id, image_data) function, including the image's unique identifier image_id and image_data. Then use Redis's set command to store image_id as the key and image_data as the value in Redis to implement image caching. Then, use Redis's get command to retrieve and return the image in the cache based on image_id. Correspondingly, server 400 also obtains the list all_keys of all keys (cache keys) in Redis. Then, traverse these keys and check the frequency of use of each key. If the frequency of use of a cache key does not meet the standard, use Redis's delete command to delete the image cache.

[0132] like Fig.12 As shown, in order to reduce the waiting time of the user at the display device 200, in some embodiments, the server 400 also detects the focus area of ​​the image corresponding to the media asset data. The focus area is determined according to the position of the control focus in the historical operation log, such as determining the image area that the user pays most attention to based on the user's browsing behavior (such as mouse movement, click). Then, the image corresponding to the media asset data is processed by the focus area to generate at least two block data. Then, the block data is cached to the remote dictionary service.

[0133] In some embodiments, when the server 400 performs block processing on the image corresponding to the media data, it can also perform block processing according to other principles, such as caching and loading core areas such as faces and important objects first, while background or non-critical content is loaded later. This method not only improves the loading speed, but also reduces the network bandwidth usage. Among them, the server 400 can set priorities for the block data, such as loading the core area first and loading the background later.

[0134] For example, create a CacheSystem instance that contains two dictionaries, one for storing the core area cache (core_cache) and the other for storing the non-core area cache (non_core_cache). Call the load_image_with_smart_caching function, passing in the image URL and the cache system instance. First, check whether there is a core area corresponding to the image URL in the cache system through cache_system.get_core_block(image_url). If the core area cache hits, get the core area directly from the cache and display it.

[0135] If the core area is not in the cache, download the entire image and use the split_image_into_blocks function to split the image into the core area and the non-core area. Store the core area in the cache system and load the non-core area asynchronously. Then, traverse the non-core area blocks, and for each block, check whether the non-core area corresponding to the image URL and block ID exists in the cache system. If the cache hits, the non-core area is directly obtained from the cache and displayed; if the cache misses, the non-core area is downloaded and cached, and then displayed.

[0136] The split_image_into_blocks function uses the detect_core_content function to identify the core areas (such as faces, objects, etc.) in the image. Based on the coordinates of the core area, the extract_region function is used to extract the core area image blocks, and the extract_non_core_regions function is used to extract the remaining non-core area blocks. The detect_core_content function uses a pre-trained deep learning model to detect the core objects or areas in the image. The extract_region function extracts specific image blocks from the image based on the given region coordinates. The extract_non_core_regions function is responsible for processing the remaining part and splitting it into non-core area blocks. The download_and_cache_non_core_block function is responsible for downloading the non-core area blocks and storing them in Redis.

[0137] Accordingly, in some embodiments, when the server 400 sends the second image to the display device 200, if the second image is block data, the server 400 can send the block data to the display device 200 in sequence, so that the display device 200 can load and display the second image in blocks. Since the second image is processed in blocks according to the focus area, the display device 200 can first display the area of ​​the second image that the user is more concerned about, and then display other areas, thereby shortening the user's waiting time. That is, the intelligent block caching technology based on image content in the embodiment of the present application allows the user to see the core content of the image faster by loading the core area first.

[0138] In the embodiment of the present application, the program steps executed by the server 400 may also use other architectures to implement similar functions. For example, all or part of the processes described in the above embodiment may be completed by the display device 200, which will not be described in detail here.

[0139] Based on the above embodiments, some embodiments of the present application further provide a display device 200, including a display 260, a communication device 220 and a controller 250. The display 260 is configured to display a user interface, and the communication device 220 is configured to establish communication with a server; Fig.13 As shown, the controller 250 is configured to perform the following program steps:

[0140] S1301: In response to an interface display instruction input by a user, detecting a type parameter and a data source identifier associated with the interface display instruction;

[0141] S1302: Generate an image acquisition request according to the type parameter and the data source identifier;

[0142] S1303: Sending the image acquisition request to the server;

[0143] S1304: Receive a second image sent by the server, where the second image is an image sent by the server in response to the image acquisition request, and the second image is an image generated by cropping the first image according to the key area. The first image is an image acquired according to the type parameter and the data source identifier, and the key area is identified in the first image according to cropping parameters and a region extraction algorithm. The cropping parameters are parameters generated according to the data source identifier and the original size of the first image, and the cropping parameters include a target size and a target aspect ratio. The image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio.

[0144] S1305: Control the display to display the second image in a preset display area of ​​the user interface.

[0145] Based on the above server 400 and display device 200, some implementations of the present application also provide a method for dynamic cropping of an image, such as Figure 6 As shown, the method may include the following steps:

[0146] S601: receiving an image acquisition request sent by a display device, wherein the image acquisition request includes a type parameter and a data source identifier;

[0147] S602: In response to the image acquisition request, acquire a first image according to the type parameter and the data source identifier;

[0148] S603: Generate cropping parameters according to the data source identifier and the original size of the first image, where the cropping parameters include a target size and a target aspect ratio;

[0149] S604: identifying a key area of ​​the first image according to the cropping parameters and the area extraction algorithm;

[0150] S605: Cropping the first image according to the key area to generate a second image; the image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio;

[0151] S606: Send the second image to the display device, so that the display device displays the second image in a preset display area.

[0152] It can be known from the above technical aspects that some embodiments of the present application provide a server, a display device, and a dynamic cropping method for an image. The method can receive an image acquisition request sent by the display device 200, and the image acquisition request includes a type parameter and a data source identifier. Then, in response to the image acquisition request, a first image is acquired according to the type parameter and the data source identifier, and cropping parameters are generated according to the data source identifier and the original size of the first image. Among them, the cropping parameters include a target size and a target aspect ratio. According to the cropping parameters and the area extraction algorithm, the key area of ​​the first image is identified, and then the first image is cropped according to the key area to generate a second image. The second image is sent to the display device so that the display device displays the second image in a preset display area. The method can dynamically generate an image adapted to the display device 200, which can improve the image display effect of the display device 200 and enhance the user experience.

[0153] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0154] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0156] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A server, characterized in that: include: A communication module, configured to establish communication with a display device; The control module is configured as follows: receiving an image acquisition request sent by the display device, wherein the image acquisition request includes a type parameter and a data source identifier; In response to the image acquisition request, acquiring a first image according to the type parameter and the data source identifier; generating cropping parameters according to the data source identifier and the original size of the first image, the cropping parameters including a target size and a target aspect ratio; identifying a key area of ​​the first image according to the cropping parameters and the area extraction algorithm; Cropping the first image according to the key area to generate a second image; The image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio; The second image is sent to the display device, so that the display device displays the second image in a preset display area.

2. The server according to claim 1, characterized in that: The control module executes acquiring the first image according to the type parameter and the data source identifier, and is specifically configured as follows: Acquire reference data according to the type parameter and the data source identifier, the reference data including media asset data and historical data, the media asset data including media asset ID, media asset tag, media asset title and media asset description information, and the historical data including at least one of viewing history, click record and rating record; Extracting user tags from the historical data; Converting the media asset data into a target matrix, and converting the user tag into a target vector, wherein the target matrix is ​​a matrix generated based on word frequency and inverse document frequency, the target matrix includes a reference vector of the media asset tag based on word frequency and inverse document frequency, and the target vector is a vector generated based on word frequency and inverse document frequency; Calculating cosine similarity according to the target matrix and the target vector, the cosine similarity being the similarity between the target vector and the reference vector; Acquire the target media asset data with the highest cosine similarity; The image corresponding to the target media asset data is output through a recommendation function to obtain the first image.

3. The server according to claim 2, characterized in that: After the control module executes the step of acquiring reference data according to the type parameter and the data source identifier, the control module is further configured to: Loading a pre-trained image recognition model, wherein the image recognition model includes a convolutional neural network; Inputting the media data into the image recognition model so that the image recognition model outputs a classification prediction result of the media data; If the classification prediction result is in the preset screening classification list, the media asset data is deleted; If the category prediction result is not in the filtered category list, the media asset data is retained.

4. The server according to claim 1, characterized in that: The control module generates cropping parameters according to the data source identifier and the original size of the first image, and is specifically configured as follows: Reading the original size of the first image according to the uniform resource locator of the first image; Calculating an original aspect ratio according to the original size; Read the target aspect ratio corresponding to the data source identifier; Calculating a target size of the second image according to the target aspect ratio and the original aspect ratio; The cropping parameters are set by the target aspect ratio and the target size.

5. The server according to claim 1, characterized in that: The control module identifies the key area of ​​the first image according to the cropping parameters and the area extraction algorithm, and is specifically configured as follows: Detecting key features of the first image based on the region extraction algorithm; Acquire a main area of ​​the key feature in the first image; The key area is calculated according to the cropping parameters and the main area, the key area includes the main area, and the size of the key area is equal to the target size.

6. The server according to claim 1, characterized in that: The control module is further configured to: Connect to remote dictionary service; Creating a cache function based on the remote dictionary service; caching the media data by means of the cache function to generate a cache key for the media data, wherein the cache key is associated with an image identifier of an image corresponding to the media data; Traverse the usage frequency of the cache key; If the usage frequency is less than or equal to the frequency threshold, the cache key is deleted; If the usage frequency is greater than the frequency threshold, the cache key is retained to generate local cache data.

7. The server according to claim 6, characterized in that: The control module executes acquiring the first image according to the type parameter and the data source identifier, and is specifically configured as follows: Acquire a target identifier according to the type parameter and the data source identifier; Query the local cache data for an image identifier; If the image identifier of the local cache data includes the target identifier, an image corresponding to the target identifier is extracted from the local cache data to obtain the first image.

8. The server according to claim 6, characterized in that: The control module executes caching of media asset data through the caching function, and is specifically configured as follows: Detecting a focus area of ​​an image corresponding to the media asset data, wherein the focus area is determined according to a position of a control focus in a historical operation log; Performing block processing on the image corresponding to the media asset data through the focus area to generate at least two block data; The block data is cached in the remote dictionary service.

9. A display device, characterized in that: include: a display configured to display a user interface; a communication device configured to establish communication with a server; The controller is configured as: In response to an interface display instruction input by a user, detecting a type parameter and a data source identifier associated with the interface display instruction; Generate an image acquisition request according to the type parameter and the data source identifier; Sending the image acquisition request to the server; receiving a second image sent by the server, the second image being an image sent by the server in response to the image acquisition request, the second image being an image generated by cropping the first image according to the key area, the first image being an image acquired according to the type parameter and the data source identifier, the key area being identified in the first image according to cropping parameters and an area extraction algorithm, the cropping parameters being parameters generated according to the data source identifier and an original size of the first image, the cropping parameters including a target size and a target aspect ratio, the image size of the second image being the target size, and the image aspect ratio of the second image being the target aspect ratio; The display is controlled to display the second image in a preset display area of ​​the user interface.

10. A method for dynamic cropping of an image, characterized in that: include: Receiving an image acquisition request sent by a display device, wherein the image acquisition request includes a type parameter and a data source identifier; In response to the image acquisition request, acquiring a first image according to the type parameter and the data source identifier; generating cropping parameters according to the data source identifier and the original size of the first image, the cropping parameters including a target size and a target aspect ratio; identifying a key area of ​​the first image according to the cropping parameters and the area extraction algorithm; Cropping the first image according to the key area to generate a second image; The image size of the second image is the target size, and the image aspect ratio of the second image is the target aspect ratio; The second image is sent to the display device, so that the display device displays the second image in a preset display area.

Images