Processing method and device of visual image transmission, electronic equipment and storage medium

Abstract

The application discloses a kind of visual image transmission processing method and its device, electronic equipment and storage medium, it is related to the field of financial technology or other related fields, wherein the processing method comprises: receiving image information file and attention information file, determine current throughput rate, in the case where current throughput rate is less than rectangular block code rate, based on playing fragment vector matrix, the rectangular block indicated by the vector value less than preset threshold is degraded, and the processing result is obtained, based on processing result and storage path, download all rectangular blocks of playing fragment.The application solves the technical problem that the image quality of the user attention area cannot be effectively controlled according to the network status in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of financial technology, in particular to a visual image transmission processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, the dynamic adaptive streaming over HTTP (Dynamic Adaptive Streaming over HTTP, DASH for short) is one of the most effective solutions to ensure the high-quality and reliable transmission of streaming media. It can solve the problem of limited bandwidth encountered in the transmission process of streaming media. When the network bandwidth capacity is insufficient, the bit rate of the video is reduced to reduce the number of stall events in the video playback process. When the network bandwidth becomes better, the bit rate of the transmitted video is increased, thereby ensuring the reliability of data transmission and high-quality user experience.

[0003] However, although the dynamic adaptive streaming over HTTP can push the appropriate quality according to the network condition, when the network condition is not good, the quality of the entire video will also decrease, and the quality of the area expected to be watched will also decrease, which not only wastes the network bandwidth, but also reduces the user's viewing experience.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a visual image transmission processing method and device, electronic equipment and storage medium, to at least solve the technical problem that the image quality of the user's attention area cannot be effectively controlled according to the network condition in the related art.

[0006] According to an aspect of an embodiment of the present application, a visual image transmission processing method is provided, applied to a client, comprising: receiving an image information file and an attention information file, wherein the image information file at least includes a storage path of a rectangular block and a rectangular block code rate, and the attention information file at least includes a play segment vector matrix, the rectangular block being obtained by encoding the image by a server; determining a current throughput rate; in the case that the current throughput rate is less than the rectangular block code rate, performing degradation processing on the rectangular block indicated by the vector value less than a preset threshold based on the play segment vector matrix, to obtain a processing result; and downloading all rectangular blocks of a play segment based on the processing result and the storage path.

[0007] Optionally, before receiving the image information file and the attention information file, the method further comprises: receiving, by the server, an image transmission request sent by the client, wherein the image transmission request comprises at least a to-be-transmitted image identifier; determining, by the server, a target image corresponding to the to-be-transmitted image identifier; encoding, by the server, the target image to obtain an encoded image and the image information file; and obtaining, by the server, the attention information file based on all the rectangular blocks of each play segment.

[0008] Optionally, the step of encoding, by the server, the target image comprises: dividing, by the server, each image frame in the target image into a plurality of rectangular blocks; dividing, by the server, the target image into a plurality of play segments, wherein each play segment corresponds to a plurality of rectangular blocks; and encoding, by the server, each rectangular block in the play segment into a plurality of image blocks, wherein the hierarchical type of the image block comprises a base layer and an enhancement layer, the base layer is used to decode the image with the lowest quality, and the enhancement layer is used to enhance the quality of the image.

[0009] Optionally, the step of obtaining, by the server, the attention information file based on all the rectangular blocks of each play segment comprises: inputting, by the server, all the rectangular blocks of the play segment into a preset attention model to obtain an attention value of each rectangular block; generating, by the server, a play segment vector matrix based on all the attention values, wherein each vector value in the play segment vector matrix corresponds to the attention value one by one; and generating, by the server, the attention information file based on the play segment vector matrix.

[0010] Optionally, before inputting, by the server, all the rectangular blocks of the play segment into the preset attention model, the method further comprises: obtaining, by the server, a plurality of labeled rectangular block data to obtain training data; training, by the server, an initial attention model using the training data, and obtaining the preset attention model when the training is completed.

[0011] Optionally, before determining the current throughput rate, further comprising: downloading the base layer image block and the enhancement layer image block of a preset rectangular block in a preset play segment; in a case that a preset storage area length at a current download time is greater than or equal to a rectangular block buffer area length, downloading the base layer image block and the enhancement layer image block of a remaining rectangular block in the preset play segment and updating the current throughput rate, wherein the rectangular block buffer area length is a sum of a base layer buffer area length and an enhancement layer buffer area length of all the remaining rectangular blocks in the preset play segment, and the remaining rectangular blocks are rectangular blocks in the preset play segment other than the preset rectangular block; in a case that the preset storage area length at the current download time is less than the rectangular block buffer area length and greater than or equal to the base layer buffer area length of all the remaining rectangular blocks, downloading the base layer image block of the remaining rectangular block in the preset play segment and updating the current throughput rate.

[0012] Optionally, based on the processing result and the storage path, the step of downloading all the rectangular blocks of a play segment comprises: determining a download vector value of each rectangular block based on a target vector matrix in the processing result; in a case that the download vector value is a first preset value, downloading a base layer image block of the rectangular block according to the storage path; in a case that the download vector value is a second preset value, downloading a base layer image block and an enhancement layer image block of the rectangular block according to the storage path.

[0013] Optionally, after downloading all the rectangular blocks of a play segment based on the processing result and the storage path, further comprising: before playing a target play segment, checking whether a base layer image block of the target play segment exists to obtain a first checking result; in a case that the first checking result indicates that the base layer image block of the target play segment exists, checking whether an enhancement layer image block of the target play segment exists to obtain a second checking result; in a case that the second checking result indicates that the enhancement layer image block of the target play segment exists, decoding each rectangular block in the target play segment, and playing the target play segment in a case that decoding is completed.

[0014] Optionally, after checking whether a base layer image block of a target play segment exists to obtain a first checking result, further comprising: in a case that the first checking result indicates that the base layer image block of the target play segment does not exist, pausing playing the target play segment; downloading the base layer image block of the target play segment.

[0015] According to another aspect of the embodiments of the present application, a processing device for visual image transmission is also provided, which is applied to a client and includes: a receiving unit configured to receive an image information file and an attention information file, wherein the image information file at least includes storage paths of rectangular blocks and code rates of the rectangular blocks, and the attention information file at least includes a vector matrix of playing segments, the rectangular blocks being obtained by encoding an image by a server; a determining unit configured to determine a current throughput rate; a processing unit configured to, in a case that the current throughput rate is less than the code rates of the rectangular blocks, perform degradation processing on rectangular blocks indicated by vector values less than a preset threshold based on the vector matrix of the playing segments, to obtain a processing result; and a downloading unit configured to download all rectangular blocks of a playing segment based on the processing result and the storage paths.

[0016] Optionally, the processing device further includes: a first receiving module configured to, before receiving the image information file and the attention information file, receive, by the server, an image transmission request sent by the client, wherein the image transmission request at least includes an image to be transmitted; a first determining module configured to determine, by the server, a target image corresponding to the image to be transmitted; a first encoding module configured to encode, by the server, the target image to obtain an encoded image and the image information file; and a first output module configured to obtain, by the server, the attention information file based on all rectangular blocks of each playing segment.

[0017] Optionally, the first determining module includes: a first dividing sub-module configured to control the server to divide each image frame in the target image into a plurality of rectangular blocks; a second dividing sub-module configured to control the server to divide the target image into a plurality of playing segments, wherein each playing segment corresponds to a plurality of rectangular blocks; and a first encoding sub-module configured to control the server to encode each rectangular block in the playing segment into a plurality of image blocks, wherein a hierarchical type of the image blocks includes a base layer and an enhancement layer, the base layer being used to decode an image at a lowest quality, and the enhancement layer being used to enhance the quality of the image.

[0018] Optionally, the first output module includes: a first input sub-module configured to control the server to input all rectangular blocks of the playing segment to a preset attention model to obtain an attention value of each rectangular block; a first generation sub-module configured to control the server to generate the vector matrix of the playing segment based on all attention values, wherein each vector value in the vector matrix of the playing segment corresponds to one attention value; and a second generation sub-module configured to control the server to generate the attention information file based on the vector matrix of the playing segment.

[0019] Optionally, the processing device further comprises: a first acquisition module, configured to control the server to acquire a plurality of labeled rectangular block data to obtain training data before the server inputs all the rectangular blocks of the playing segment into the preset attention model; and a first training module, configured to control the server to train an initial attention model using the training data, and obtain the preset attention model if the training is completed.

[0020] Optionally, the processing device further comprises: a first download module, configured to download a base layer image block and an enhancement layer image block of a preset rectangular block in a preset playing segment before determining a current throughput rate; a second download module, configured to download the base layer image block and the enhancement layer image block of a remaining rectangular block in the preset playing segment and update the current throughput rate if a preset storage area length at a current download time is greater than or equal to a rectangular block buffer area length, wherein the rectangular block buffer area length is a sum of a base layer buffer area length and an enhancement layer buffer area length of all the remaining rectangular blocks in the preset playing segment, and the remaining rectangular blocks are rectangular blocks other than the preset rectangular block in the preset playing segment; and a third download module, configured to download the base layer image block of the remaining rectangular block in the preset playing segment and update the current throughput rate if the preset storage area length at the current download time is less than the rectangular block buffer area length and greater than or equal to the base layer buffer area length of all the remaining rectangular blocks.

[0021] Optionally, the download unit comprises: a second determination module, configured to determine a download vector value of each rectangular block based on a target vector matrix in the processing result; a fourth download module, configured to download a base layer image block of the rectangular block according to the storage path if the download vector value is a first preset value; and a fifth download module, configured to download a base layer image block and an enhancement layer image block of the rectangular block according to the storage path if the download vector value is a second preset value.

[0022] Optionally, the processing device further comprises: a first checking module, configured to check whether a base layer image block of a target playing segment exists before playing the target playing segment after downloading all rectangular blocks of the playing segment based on the processing result and the storage path to obtain a first checking result; a second checking module, configured to check whether an enhancement layer image block of the target playing segment exists if the first checking result indicates that the base layer image block of the target playing segment exists to obtain a second checking result; and a first decoding module, configured to decode each rectangular block in the target playing segment if the second checking result indicates that the enhancement layer image block of the target playing segment exists, and play the target playing segment if the decoding is completed.

[0023] Optionally, the processing device further comprises: a first pause module, configured to, after checking whether the target playing segment has the base layer image block, obtaining a first checking result, pause playing the target playing segment when the first checking result indicates that the target playing segment does not have the base layer image block; and a sixth downloading module, configured to download the base layer image block of the target playing segment.

[0024] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, which comprises a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to perform the processing method of visual image transmission when the computer program runs.

[0025] According to another aspect of the embodiments of the present application, an electronic device is provided, which comprises one or more processors and a memory, and the memory is configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the processing method of visual image transmission.

[0026] In the present disclosure, the image information file and the attention information file are received, the current throughput rate is determined, and in the case that the current throughput rate is less than the rectangular block code rate, the rectangular block indicated by the vector value less than the preset threshold is degraded based on the playing segment vector matrix to obtain a processing result, and all rectangular blocks of the playing segment are downloaded based on the processing result and the storage path. In the present disclosure, the client can first determine the current throughput rate, and then compare the current throughput rate with the rectangular block code rate in the image information file sent by the server. If the current throughput rate is less than the rectangular block code rate, the user-unattended area (i.e., the rectangular block indicated by the vector value less than the preset threshold) needs to be degraded according to the playing segment vector matrix in the attention information file sent by the server. Then, all rectangular blocks of the playing segment are downloaded according to the processing result and the storage path of the rectangular block in the image information file for playing. The image quality of the user-attended area can be controlled according to the network condition, the user image viewing experience is improved, and the technical problem that the image quality of the user-attended area cannot be effectively controlled according to the network condition in the related art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0028] Figure 1 is a flowchart of an optional processing method of visual image transmission according to an embodiment of the present application;

[0029] Figure 2 is a schematic diagram of an optional visual image transmission optimization architecture according to an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of an optional Tile encoding result according to an embodiment of the present application;

[0031] Figure 4 is a schematic diagram of an optional SVC encoding result according to an embodiment of the present application;

[0032] Figure 5 is a schematic diagram of an optional buffer length according to an embodiment of the present application;

[0033] Figure 6 is a schematic diagram of an optional visual image transmission processing device according to an embodiment of the present application;

[0034] Figure 7 is a hardware structure block diagram of an electronic device (or mobile device) for a processing method of visual image transmission according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0037] In order to facilitate those skilled in the art to understand the present application, the following explains some terms or names involved in each embodiment of the present application:

[0038] Tile coding: that is, block coding, the plane video is divided into equal size sub-rectangular blocks according to the grid, each sub-rectangular block is called tile, and each tile can be independently decoded and played.

[0039] SVC coding: that is, scalable video coding, the video is coded into multiple quality levels with dependency relationship.

[0040] It should be noted that the processing method and device for visual image transmission in the present disclosure can be used in the field of financial technology when processing visual image transmission, and can also be used in any field other than the field of financial technology when processing visual image transmission. The application field of the processing method and device for visual image transmission in the present disclosure is not limited.

[0041] It should be noted that the related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal. For example, an interface is provided between the system and the related user or institution, and before obtaining the related information, the interface needs to send an acquisition request to the aforementioned user or institution, and after receiving the consent information fed back by the aforementioned user or institution, the related information is acquired.

[0042] The following embodiments of the present application can be applied to various systems / applications / devices for processing visual image transmission. The present application provides a method for incrementally improving the image quality of the user's attention area according to the network condition, which can overcome the problem of low user viewing experience caused by low video resolution in the case of insufficient network transmission capacity, and improve the user's viewing experience.

[0043] The present application will be described in detail below in conjunction with various embodiments.

[0044] Embodiment one

[0045] According to the embodiments of the present application, an embodiment of a processing method for visual image transmission is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0046] Figure 1 is a flowchart of an optional processing method for visual image transmission according to the embodiments of the present application, like Figure 1As shown, the method comprises the following steps:

[0047] In step S101, an image information file and an attention information file are received, wherein the image information file at least comprises a storage path of a rectangular block and a rectangular block code rate, and the attention information file at least comprises a play segment vector matrix, and the rectangular block is obtained by encoding the image by a server.

[0048] In step S102, a current throughput rate is determined.

[0049] In step S103, in a case where the current throughput rate is less than the rectangular block code rate, a rectangular block indicated by a vector value less than a preset threshold is degraded based on the play segment vector matrix to obtain a processing result.

[0050] In step S104, all rectangular blocks of a play segment are downloaded based on the processing result and the storage path.

[0051] Through the above steps, the image information file and the attention information file can be received, the current throughput rate can be determined, in a case where the current throughput rate is less than the rectangular block code rate, the rectangular block indicated by the vector value less than the preset threshold is degraded based on the play segment vector matrix to obtain the processing result, and all rectangular blocks of the play segment are downloaded based on the processing result and the storage path. In the embodiment of the present application, the client can first determine the current throughput rate, then compare the current throughput rate with the rectangular block code rate in the image information file sent by the server, if the current throughput rate is less than the rectangular block code rate, then the user-unattended area (i.e., the rectangular block indicated by the vector value less than the preset threshold) needs to be degraded according to the play segment vector matrix in the attention information file sent by the server, and then all rectangular blocks of the play segment are downloaded according to the processing result and the storage path of the rectangular block in the image information file for playing, so that the image quality of the user-attended area can be controlled according to the network condition, the user image viewing experience is improved, and the technical problem that the image quality of the user-attended area cannot be effectively controlled according to the network condition in the related art is solved.

[0052] The embodiment of the present application will be described in detail below in combination with the above steps. Each of the following steps can be applied to the client.

[0053] The embodiment can be based on a client-server (C / S) mode, two systems are respectively designed on the client and the server, and appropriate video is pushed to the client according to the network condition, so that the user obtains better viewing experience.

[0054] Figure 2 is a schematic diagram of an optional visual image transmission optimization architecture according to the embodiment of the present application, like Figure 2As shown, the video server (i.e. server side) is connected with the client through the network, the server side completes the encoding of the video (i.e. the server side first encodes the video by tile, then obtains a plurality of rectangular blocks, and then divides the video into a plurality of time-equal playing segments (seg0,..., seg n-1 ), and then encodes each rectangular block in the playing segment by SVC, encodes each rectangular block into three layers of video blocks Chunk of L0, L1 and L2, stores the encoded video file (i.e. Video) and the corresponding MPD file (the MPD file is used to describe the related information of the DASH server side streaming video slice), and stores the video attention area file. The main task of the client is to initialize the video request, adjust the quality of the user attention area according to the video attention area file, adjust the quality of the downloaded video file and the video playing quality according to the video download throughput change, and the specific method can be that the network interface receives the MPD file and the video attention area file sent by the server side, the adaptive adjustment is performed through the bandwidth prediction module, the attention area selection module and the buffer state control adaptive logic module of the buffer, the corresponding video block Chunk is downloaded to the buffer through the adjustment decision calling network interface, and finally the video is played by the playing module.

[0055] An optional embodiment, before receiving the image information file and the attention information file, further comprising: the server side receives the image transmission request sent by the client, wherein the image transmission request at least includes: a to-be-transmitted image identifier; the server side determines a target image corresponding to the to-be-transmitted image identifier; the server side encodes the target image to obtain an encoded image and an image information file; and the server side obtains the attention information file based on all rectangular blocks of each playing segment.

[0056] In the embodiment of the application, the server side can first receive the image transmission request sent by the client (the image transmission request includes a to-be-transmitted image identifier (i.e. a unique identifier of the image)), then determine a target image corresponding to the to-be-transmitted image identifier according to the image transmission request, and then encode the target image to obtain an encoded image and a corresponding image information file (i.e. an MPD file, which is a file for recording image information generated in the process of encoding the image, for example, image attribute information, image storage path, etc.), and can also obtain an attention information file according to all rectangular blocks of a playing segment. The attention information file includes a playing segment vector matrix corresponding to each playing segment (i.e. according to how much content a rectangular block contains, the attention value of the rectangular block is determined, and the higher the attention value, the more content the rectangular block contains, and the more worthy of being paid attention to by the user).

[0057] Optionally, the step of encoding the target video by the server comprises: dividing each video frame in the target video into a plurality of rectangular blocks by the server; dividing the target video into a plurality of playing segments by the server, wherein each playing segment corresponds to a plurality of rectangular blocks; and encoding each rectangular block in the playing segment into a plurality of video blocks by the server, wherein the hierarchical type of the video block comprises: a base layer and an enhancement layer, the base layer is used to decode the video at the lowest quality, and the enhancement layer is used to enhance the quality of the video.

[0058] In the embodiment of the present application, the server can first encode the target video, specifically: the video frame (i.e. video frame) can be divided into a plurality of rectangular blocks (i.e. the server divides each video frame in the target video into a plurality of rectangular blocks) of equal size in space, and the tiles are numbered in the order from left to right and from top to bottom. In the time dimension, the video can be divided into a plurality of time-equal playing segments (i.e. the server divides the target video into a plurality of playing segments, and each playing segment corresponds to a plurality of rectangular blocks). For each tile in each segment, it is encoded into a plurality of video blocks (i.e. the server encodes each rectangular block in the playing segment into a plurality of video blocks) in the hierarchical dimension, and the hierarchical type of the video block (i.e. video block) includes: a base layer and an enhancement layer, the base layer is used to decode the video at the lowest quality, and the enhancement layer is used to enhance the quality of the video.

[0059] In the embodiment, the video block chunk is the smallest unit of video playback requested by the client.

[0060] In the embodiment, for a two-dimensional plane video, it can be divided into M tiles according to the longitude and N tiles according to the latitude, so there are M×N tiles in total. Figure 3 is a schematic diagram of an optional Tile encoding result according to an embodiment of the present application, as shown in Figure 3 A certain two-dimensional plane video is divided into 6×4 tiles and numbered in the order from left to right and from top to bottom, i.e. tile_1 to tile_24.

[0061] In the embodiment, in the encoding stream, there is a mandatory base layer (Base Layer, BL) and a plurality of optional enhancement layers (Enhancement Layer, EL). Figure 4 is a schematic diagram of an optional SVC encoding result according to an embodiment of the present application, as shown in Figure 4As shown, the video is encoded into 3 layers (i.e. base layer BL0, enhancement layer EL1, and enhancement layer EL2). Among them, BL is an independent unit, and the video can be decoded at the lowest quality. EL is used to enhance the quality of the video. The video layers are highly interdependent. BL is necessary for decoding any video. EL1 can only be used when the corresponding BL already exists. Similarly, EL2 can only be used when the BL and EL1 already exist, and so on.

[0062] Optionally, the step of obtaining the attention information file based on all the rectangular blocks of each playing segment by the server comprises: inputting all the rectangular blocks of the playing segment into a preset attention model by the server to obtain an attention value of each rectangular block; generating a playing segment vector matrix based on all the attention values by the server, wherein each vector value in the playing segment vector matrix corresponds to an attention value one by one; and generating the attention information file based on the playing segment vector matrix by the server.

[0063] In the embodiment of the application, the server can generate an attention information file (i.e. a video attention area file), specifically: the rectangular blocks of each segment divided can be analyzed, if the rectangular block has more content, it is inclined to download to the highest layer of the video block, and the attention value can be set to the highest (such as 2); if the rectangular block has less content or is near the video block with more content, it can be downloaded to the lower enhancement layer of the video block, and the attention value can be set to be higher (such as 1), which is used to prevent the video quality from decreasing too fast; if the rectangular block has no content, the attention value can be set to the lowest (such as 0), so only the base layer video block needs to be downloaded (i.e. inputting all the rectangular blocks of the playing segment into a preset attention model by the server to obtain an attention value of each rectangular block). Then a vector set (i.e. generating a playing segment vector matrix based on all the attention values by the server, wherein each vector value in the playing segment vector matrix corresponds to an attention value one by one) can be obtained, for example, the playing segment vector matrix is:

[0064]

[0065] Then, the attention information file can be generated according to the playing segment vector matrix.

[0066] Optionally, before inputting all the rectangular blocks of the playing segment into the preset attention model by the server, it further comprises: obtaining a plurality of labeled rectangular block data by the server to obtain training data; training an initial attention model by the server using the training data, and obtaining the preset attention model when the training is completed.

[0067] In the embodiment of the present application, the preset attention model is used to identify the content contained in the rectangular block and output the corresponding attention value. Therefore, the embodiment can train the preset attention model using training data, specifically: a plurality of labeled rectangular block data (i.e., rectangular blocks of pre-labeled information, the labeled information being the attention value, and the higher the attention value, the more content the rectangular block contains) can be obtained to obtain training data, and then the initial attention model is trained using the training data until the model converges to obtain the final preset attention model (i.e., the preset attention model is obtained when the training is completed).

[0068] In step S101, the image information file and the attention information file are received, wherein the image information file at least includes the storage path of the rectangular block and the rectangular block code rate, and the attention information file at least includes the play segment vector matrix. The rectangular block is obtained by encoding the image by the server.

[0069] In the embodiment of the present application, the client can first receive the image information file and the attention information file sent by the server, the image information file includes the storage path (i.e., the storage location of the rectangular block) and the rectangular block code rate (i.e., the rate of downloading the rectangular block) and other information, and the rectangular block is obtained by encoding the image by the server. The attention information file includes the play segment vector matrix and other information (i.e., the attention information file is used to indicate whether the rectangular block is the rectangular block of the user attention area).

[0070] Optionally, before determining the current throughput rate, it further includes: downloading the base layer image block and the enhancement layer image block of the preset rectangular block in the preset play segment; in the case that the preset storage area length at the current download time is greater than or equal to the rectangular block buffer area length, downloading the base layer image block and the enhancement layer image block of the remaining rectangular block in the preset play segment and updating the current throughput rate, wherein the rectangular block buffer area length is the sum of the base layer buffer area length and the enhancement layer buffer area length of all the remaining rectangular blocks in the preset play segment, and the remaining rectangular block is the rectangular block in the preset play segment except the preset rectangular block; in the case that the preset storage area length at the current download time is less than the rectangular block buffer area length and greater than or equal to the base layer buffer area length of all the remaining rectangular blocks, downloading the base layer image block of the remaining rectangular block in the preset play segment and updating the current throughput rate.

[0071] In the embodiment of the present application, the client needs to be initialized to download. In order to estimate the size of the download bandwidth, the client needs to download the base layer and the enhancement layer video blocks of the tile in front of the first playing segment (i.e. the preset playing segment, which is the first segment of the image to be played) at the beginning of the playing, i.e. the base layer and the enhancement layer image blocks of the preset tile in the preset playing segment, so that the playing segment can be played with high quality video at the beginning of the playing. When the length of the buffer at the initialization download time (i.e. the preset storage length, which is used to store the image blocks downloaded by the client) is greater than or equal to the tile buffer length (which is the sum of the base layer buffer length and the enhancement layer buffer length of all the remaining tiles in the preset playing segment, and the remaining tile is the tile in the preset playing segment except the preset tile), the video blocks of all the layers of all the remaining tiles at this time are downloaded, put into the buffer and the average throughput is updated (i.e. in the case that the preset storage length at the current download time is greater than or equal to the tile buffer length, the base layer and the enhancement layer image blocks of the remaining tiles in the preset playing segment are downloaded, and the current throughput is updated). When the length of the buffer at the initialization download time is less than the tile buffer length and greater than or equal to the base layer buffer length of all the remaining tiles, the video blocks of the base layer of all the remaining tiles at this time are downloaded, put into the buffer and the average throughput is updated (i.e. in the case that the preset storage length at the current download time is less than the tile buffer length and greater than or equal to the base layer buffer length of all the remaining tiles, the base layer image blocks of the remaining tiles in the preset playing segment are downloaded, and the current throughput is updated).

[0072] In the embodiment of the present application, the base layer buffer length is greater than the enhancement layer buffer length for the buffer, because the base layer is necessary for playing the video, and the enhancement layer is used to enhance the quality of the video. Figure 5 is a schematic diagram of an optional buffer length according to the embodiment of the present application, as shown in Figure 5 The base layer (BL0) buffer length can be set to 5, and the enhancement layer (i.e. EL1 and EL2) buffer length can be set to 2.

[0073] Step S102, determine the current throughput.

[0074] In the embodiment of the present application, the current throughput can be calculated according to the file size of the downloaded tile and the time length of the download.

[0075] Step S103, in the case that the current throughput is less than the tile code rate, the tile indicated by the vector value less than the preset threshold is downgraded based on the playing segment vector matrix, and the processing result is obtained.

[0076] In the embodiment of the present application, the client can adaptively download, specifically, the client can compare the information in the MPD file (i.e., the rectangular block code rate information in the image information file) with the currently calculated download throughput (i.e., the current throughput rate), if the current download throughput is less than the code rate of the video, the unimportant video area (i.e., the rectangular block with a vector value less than a preset threshold in the playback segment vector matrix) will be downloaded in a degraded manner (i.e., the enhancement layer is reduced in level or only the base layer is downloaded) (i.e., in the case that the current throughput rate is less than the rectangular block code rate, based on the playback segment vector matrix, the rectangular block indicated by the vector value less than the preset threshold is processed in a degraded manner to obtain a processing result, the processing result is used to record the level of the image block that each rectangular block can download), so that the download throughput is not less than the code rate of the video. And during this period, in order to reduce the problem that the bandwidth estimation value is too low, every time a higher level video block is downloaded.

[0077] In step S104, all rectangular blocks of the playback segment are downloaded based on the processing result and the storage path.

[0078] Optionally, the step of downloading all rectangular blocks of the playback segment based on the processing result and the storage path comprises: determining the download vector value of each rectangular block based on the target vector matrix in the processing result; in the case that the download vector value is a first preset value, downloading the base layer image block of the rectangular block according to the storage path; in the case that the download vector value is a second preset value, downloading the base layer image block and the enhancement layer image block of the rectangular block according to the storage path.

[0079] In the embodiment of the present application, the client can download all rectangular blocks of the playback segment according to the processing result and the storage path of the rectangular block in the image information file, specifically, the download vector value (i.e., the attention value, used to indicate the level of the image block that the rectangular block can download) of each rectangular block is determined according to the target vector matrix (i.e., the vector matrix composed of the attention values of each rectangular block after processing) in the processing result, if the download vector value is a first preset value (i.e., the attention value is small, for example, 0 or 1), the base layer image block of the rectangular block is downloaded according to the storage path; if the download vector value is a second preset value (i.e., the attention value is large, for example, 2), the base layer image block and the enhancement layer image block of the rectangular block are downloaded according to the storage path.

[0080] Optionally, after downloading all the rectangular blocks of the playing segment based on the processing result and the storage path, the method further comprises: before playing the target playing segment, checking whether the base layer image block of the target playing segment exists to obtain a first checking result; in the case that the first checking result indicates that the base layer image block of the target playing segment exists, checking whether the enhancement layer image block of the target playing segment exists to obtain a second checking result; in the case that the second checking result indicates that the enhancement layer image block of the target playing segment exists, decoding each rectangular block in the target playing segment, and playing the target playing segment in the case that the decoding is completed.

[0081] In the embodiment of the present application, the client can adaptively play, specifically: for the video blocks that have been downloaded, when fast playing reaches the tth segment (i.e. the target playing segment), it is checked whether the base layer video block of the tth segment exists (i.e. before playing the target playing segment, it is checked whether the base layer image block of the target playing segment exists to obtain a first checking result), if it exists, it is further checked whether the enhancement layer of L1 exists, if it also exists, it is further checked whether the enhancement layer of L2 exists (if the enhancement layer of L1 does not exist, it will not be checked any more), and then each tile is parsed and played (i.e. in the case that the first checking result indicates that the base layer image block of the target playing segment exists, it is checked whether the enhancement layer image block of the target playing segment exists to obtain a second checking result, and in the case that the second checking result indicates that the enhancement layer image block of the target playing segment exists, each rectangular block in the target playing segment is decoded, and the target playing segment is played in the case that the decoding is completed).

[0082] Optionally, after checking whether the base layer image block of the target playing segment exists to obtain a first checking result, the method further comprises: in the case that the first checking result indicates that the base layer image block of the target playing segment does not exist, pausing playing the target playing segment; and downloading the base layer image block of the target playing segment.

[0083] In the embodiment of the present application, if the base layer video block of the tth segment does not exist, it is paused (i.e. in the case that the first checking result indicates that the base layer image block of the target playing segment does not exist, playing the target playing segment is paused), and the client adaptive downloading module is requested to download the base layer video block of the region, and after the downloading is completed, the playing can be continued (i.e. the base layer image block of the target playing segment is downloaded).

[0084] In the embodiment of the present application, by using the SVC encoding and the block encoding technology, the whole video is divided into multiple video regions for transmission, and the video of each region is divided into multiple levels, so that in the case that the network transmission capacity is insufficient, the video quality of the user concerned region can be controlled according to the network condition, thereby improving the user video watching experience.

[0085] The following will be described in detail in combination with another embodiment.

[0086] Embodiment Two

[0087] The processing device for visual image transmission provided in this embodiment includes a plurality of implementation units, each of which corresponds to each implementation step in Embodiment One.

[0088] Figure 6 is a schematic diagram of an optional processing device for visual image transmission according to an embodiment of the present application, as shown in the figure, the processing device can include: a receiving unit 60, a determining unit 61, a processing unit 62, a downloading unit 63, wherein, Figure 6

[0089] The receiving unit 60 is configured to receive an image information file and an attention information file, wherein the image information file at least includes a storage path of a rectangular block and a rectangular block code rate, and the attention information file at least includes a play segment vector matrix, and the rectangular block is obtained by encoding the image by a server;

[0090] The determining unit 61 is configured to determine a current throughput rate;

[0091] The processing unit 62 is configured to, in a case where the current throughput rate is less than the rectangular block code rate, perform degradation processing on a rectangular block indicated by a vector value less than a preset threshold based on the play segment vector matrix, to obtain a processing result;

[0092] The downloading unit 63 is configured to download all rectangular blocks of a play segment based on the processing result and the storage path.

[0093] ​The processing device can receive the image information file and the attention information file through the receiving unit 60, determine the current throughput rate through the determining unit 61, perform degradation processing on the rectangular block indicated by the vector value less than the preset threshold based on the playback segment vector matrix to obtain a processing result through the processing unit 62 in the case that the current throughput rate is less than the rectangular block code rate, and download all the rectangular blocks of the playback segment based on the processing result and the storage path through the downloading unit 63. In the embodiment of the present application, the client can first determine the current throughput rate, and then compare the current throughput rate with the rectangular block code rate in the image information file sent by the server. If the current throughput rate is less than the rectangular block code rate, the user-unattended area (i.e., the rectangular block indicated by the vector value less than the preset threshold) needs to be degraded according to the playback segment vector matrix in the attention information file sent by the server, and then all the rectangular blocks of the playback segment are downloaded according to the processing result and the storage path of the rectangular block in the image information file for playing. The image quality of the user-attended area can be controlled according to the network condition, the user image viewing experience is improved, and the technical problem that the image quality of the user-attended area cannot be effectively controlled according to the network condition in the related art is solved.

[0094] Optionally, the processing device further comprises: a first receiving module, configured to receive, by the server, an image transmission request sent by the client before receiving the image information file and the attention information file, wherein the image transmission request at least comprises: a to-be-transmitted image identifier; a first determining module, configured to determine, by the server, a target image corresponding to the to-be-transmitted image identifier; a first encoding module, configured to encode, by the server, the target image to obtain an encoded image and the image information file; and a first output module, configured to obtain, by the server, the attention information file based on all the rectangular blocks of each playback segment.

[0095] Optionally, the first determining module comprises: a first division sub-module, configured to control the server to divide each image frame in the target image into a plurality of rectangular blocks; a second division sub-module, configured to control the server to divide the target image into a plurality of playback segments, wherein each playback segment corresponds to a plurality of rectangular blocks; and a first encoding sub-module, configured to control the server to encode each rectangular block in the playback segment into a plurality of image blocks, wherein the hierarchical type of the image block comprises: a base layer and an enhancement layer, the base layer is used to decode the image at the lowest quality, and the enhancement layer is used to enhance the quality of the image.

[0096] Optionally, the first output module comprises: a first input submodule configured to control the server to input all the rectangular blocks of the playing segment into a preset attention model to obtain an attention value of each rectangular block; a first generation submodule configured to control the server to generate a playing segment vector matrix based on all the attention values, wherein each vector value in the playing segment vector matrix corresponds to an attention value; and a second generation submodule configured to control the server to generate the attention information file based on the playing segment vector matrix.

[0097] Optionally, the processing device further comprises: a first acquisition module configured to control the server to acquire a plurality of labeled rectangular block data to obtain training data before the server inputs all the rectangular blocks of the playing segment into the preset attention model; and a first training module configured to control the server to train the initial attention model using the training data, and obtain the preset attention model if the training is completed.

[0098] Optionally, the processing device further comprises: a first download module configured to download the base layer image block and the enhancement layer image block of the preset rectangular block in the preset playing segment before determining the current throughput rate; a second download module configured to download the base layer image block and the enhancement layer image block of the remaining rectangular blocks in the preset playing segment and update the current throughput rate if the preset storage area length at the current download time is greater than or equal to the rectangular block buffer area length, wherein the rectangular block buffer area length is the sum of the base layer buffer area length and the enhancement layer buffer area length of all the remaining rectangular blocks in the preset playing segment, and the remaining rectangular blocks are rectangular blocks other than the preset rectangular block in the preset playing segment; and a third download module configured to download the base layer image block of the remaining rectangular blocks in the preset playing segment and update the current throughput rate if the preset storage area length at the current download time is less than the rectangular block buffer area length and greater than or equal to the base layer buffer area length of all the remaining rectangular blocks.

[0099] Optionally, the download unit comprises: a second determination module configured to determine a download vector value of each rectangular block based on the target vector matrix in the processing result; a fourth download module configured to download the base layer image block of the rectangular block according to the storage path if the download vector value is a first preset value; and a fifth download module configured to download the base layer image block and the enhancement layer image block of the rectangular block according to the storage path if the download vector value is a second preset value.

[0100] Optionally, the processing device further comprises: a first checking module, configured to check whether the base layer image block of the target play segment exists before playing the target play segment after downloading all the rectangular blocks of the target play segment based on the processing result and the storage path, to obtain a first checking result; a second checking module, configured to check whether the enhancement layer image block of the target play segment exists in a case where the first checking result indicates that the base layer image block of the target play segment exists, to obtain a second checking result; and a first decoding module, configured to decode each rectangular block in the target play segment in a case where the second checking result indicates that the enhancement layer image block of the target play segment exists, and play the target play segment in a case where the decoding is completed.

[0101] Optionally, the processing device further comprises: a first pausing module, configured to pause playing the target play segment in a case where the first checking result indicates that the base layer image block of the target play segment does not exist after checking whether the base layer image block of the target play segment exists to obtain the first checking result; and a sixth downloading module, configured to download the base layer image block of the target play segment.

[0102] The processing device described above can further comprise a processor and a memory, and the receiving unit 60, the determining unit 61, the processing unit 62, the downloading unit 63 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0103] The processor described above comprises a core, and the core calls the corresponding program units from the memory. The core can be one or more, and the core parameters are adjusted to download all the rectangular blocks of the play segment based on the processing result and the storage path.

[0104] The memory described above can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0105] The application further provides a computer program product adapted to execute the program of the following method steps when executed on a data processing device: receiving an image information file and an attention information file, determining a current throughput rate, performing degradation processing on a rectangular block indicated by a vector value less than a preset threshold based on a play segment vector matrix in a case where the current throughput rate is less than a rectangular block code rate, to obtain a processing result, and downloading all the rectangular blocks of the play segment based on the processing result and a storage path.

[0106] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, including a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to perform the processing method of visual image transmission when the computer program is executed.

[0107] According to another aspect of the embodiments of the present application, an electronic device is also provided, including one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the processing method of visual image transmission.

[0108] Figure 7 is a hardware structure block diagram of an electronic device (or a mobile device) for a processing method of visual image transmission according to an embodiment of the present application. As shown in Figure 7 , the electronic device can include one or more (as shown in Figure 7 702a, 702b, …, 702n) processors 702 (the processor 702 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 704 for storing data. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. Those skilled in the art can understand that Figure 7 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also include more or less components than those shown in Figure 7 , or have a different configuration from Figure 7 .

[0109] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0110] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0111] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other means. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0112] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0113] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0114] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0115] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for processing visual image transmission, characterized in that, Applied to the client side, including: The system receives an image information file and a attention information file. The image information file includes at least the storage path and bitrate of the rectangular blocks. The attention information file includes at least a playback segment vector matrix. The rectangular blocks are obtained by encoding the image on the server side. The playback segment vector matrix represents the attention value of each rectangular block. Determine the current throughput; If the current throughput is less than the rectangular block bitrate, based on the playback segment vector matrix, the rectangular blocks indicated by vector values ​​less than a preset threshold are downgraded to obtain the processing result. Based on the processing result and the storage path, download all rectangular blocks of the playback segment; This process, which begins before receiving image information files and attention information files, also includes: The server receives an image transmission request sent by the client, wherein the image transmission request includes at least: an image identifier to be transmitted; The server determines the target image corresponding to the image identifier to be transmitted; The server encodes the target image to obtain the encoded image and the image information file; The server obtains the attention information file based on all the rectangular blocks of each playback segment; The server encodes the target image by performing layered encoding on rectangular blocks of each image frame in the target image.

2. The processing method according to claim 1, characterized in that, The step of encoding the target image by the server includes: The server divides each image frame in the target image into multiple rectangular blocks; The server divides the target image into multiple playback segments, wherein each playback segment corresponds to multiple rectangular blocks; The server encodes each rectangular block in the playback segment into multiple image blocks, wherein the image block hierarchy includes a base layer and an enhancement layer. The base layer is used to decode the image at the lowest quality, and the enhancement layer is used to enhance the quality of the image.

3. The processing method according to claim 1, characterized in that, The step of the server obtaining the attention information file based on all rectangular blocks of each playback segment includes: The server inputs all the rectangular blocks of the playback segment into a preset attention model to obtain the attention value of each rectangular block; The server generates the playback segment vector matrix based on all attention values, wherein each vector value in the playback segment vector matrix corresponds one-to-one with the attention value; The server generates the attention information file based on the playback segment vector matrix.

4. The processing method according to claim 3, characterized in that, Before the server inputs all the rectangular blocks of the playback segment into the preset attention model, the method further includes: The server obtains multiple labeled rectangular blocks of data to get training data; The server uses the training data to train an initial attention model, and upon completion of training, obtains the preset attention model.

5. The processing method according to claim 1, characterized in that, Before determining the current throughput, the following steps are also included: Download the base layer image block and enhancement layer image block of the preset rectangular block in the preset playback clip; If the preset storage area length at the current download time is greater than or equal to the rectangular block buffer length, download the base layer image block and the enhancement layer image block of the remaining rectangular blocks in the preset playback segment, and update the current throughput. The rectangular block buffer length is the sum of the base layer buffer length and the enhancement layer buffer length of all the remaining rectangular blocks in the preset playback segment, and the remaining rectangular blocks are the rectangular blocks in the preset playback segment other than the preset rectangular blocks. If the length of the preset storage area at the current download time is less than the length of the rectangular block buffer but greater than or equal to the length of the base layer buffer of all the remaining rectangular blocks, download the base layer image block of the remaining rectangular blocks in the preset playback segment and update the current throughput.

6. The processing method according to claim 1, characterized in that, Based on the processing result and the storage path, the steps for downloading all rectangular blocks of the playback segment include: Based on the target vector matrix in the processing result, the download vector value of each rectangular block is determined; When the downloaded vector value is a first preset value, the base layer image block of the rectangular block is downloaded according to the storage path; When the download vector value is the second preset value, the base layer image block and the enhancement layer image block of the rectangular block are downloaded according to the storage path.

7. The processing method according to claim 1, characterized in that, After downloading all rectangular blocks of the playback segment based on the processing result and the storage path, the process further includes: Before playing the target playback segment, check whether the base layer image block of the target playback segment exists to obtain the first check result; If the first inspection result indicates the existence of the base layer image block of the target playback segment, check whether the enhancement layer image block of the target playback segment exists to obtain a second inspection result; If the second inspection result indicates the presence of the enhancement layer image block of the target playback segment, each of the rectangular blocks in the target playback segment is decoded, and the target playback segment is played upon completion of decoding.

8. The processing method according to claim 7, characterized in that, After checking whether the base layer image block of the target playback segment exists and obtaining the first check result, the method further includes: If the first check result indicates that the base layer image block of the target playback segment does not exist, the playback of the target playback segment is paused. Download the base layer image block of the target playback segment.

9. A processing apparatus for visual image transmission, characterized in that, Applied to the client side, including: A receiving unit is configured to receive an image information file and a attention information file, wherein the image information file includes at least: the storage path of the rectangular blocks and the bitrate of the rectangular blocks, and the attention information file includes at least: a playback segment vector matrix, wherein the rectangular blocks are obtained by the server encoding the image; and the playback segment vector matrix represents the attention value of each rectangular block. The determination unit is used to determine the current throughput. The processing unit is configured to, when the current throughput is less than the rectangular block bitrate, based on the playback segment vector matrix, downgrade the rectangular blocks indicated by vector values ​​less than a preset threshold to obtain a processing result. The download unit is used to download all rectangular blocks of the playback segment based on the processing result and the storage path; The processing apparatus further includes: a first receiving module, configured to receive an image transmission request sent by a client before receiving the image information file and the attention information file, wherein the image transmission request includes at least: an image identifier to be transmitted; a first determining module, configured to determine the target image corresponding to the image identifier to be transmitted; a first encoding module, configured to encode the target image to obtain an encoded image and an image information file; and a first output module, configured to obtain the attention information file based on all rectangular blocks of each playback segment; wherein the encoding of the target image by the server is a layered encoding of the rectangular blocks of each image frame in the target image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the visual image transmission processing method according to any one of claims 1 to 8.

11. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the visual image transmission processing method according to any one of claims 1 to 8.

Images