Multi-modal data compression method and device

Abstract

The invention discloses a multi-modal data compression method and device. The method comprises the following steps: acquiring a t-th network condition of multi-modal data at a t-th moment; determining that a compression strategy j with the minimum total cost is adopted for the multi-modal data under the t-th network condition; compressing each type of modal data in the multi-modal data by adopting a compression strategy j; wherein the total cost for the compression strategy m is determined according to the instant cost at the t-th moment and the influence cost at the t-th moment when the compression strategy m is adopted by the multi-modal data under the t-th network condition; the instant cost at the t moment represents the influence on the quality and / or transmission speed of the multi-modal data after compression; the influence cost at the t moment represents the sum of the minimum total cost at the (t-1) moment with the corresponding network state transition probability as the weight when various compression strategies are adopted under various network conditions at the (t-1) moment. By adopting the method, the compression strategy can be flexibly adjusted according to the dynamic change of the network state.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal data compression method and device. Background Art

[0002] At present, with the development of science and technology, data compression can be seen everywhere in life. For example, when holding a meeting through Tencent Conference, it is necessary to transmit audio or transmit audio and video at the same time; when watching videos through a tablet computer, it is necessary to transmit audio and video. Data is usually transmitted in a compressed state during the transmission process.

[0003] However, data transmission often depends on the support of the network, but the network condition is an unstable factor. Over a period of time, the network condition changes dynamically, and the existing data compression methods often cannot adapt to the dynamically changing network conditions. Summary of the invention

[0004] In a first aspect, the present application provides a multimodal data compression method, which can be performed by a multimodal data compression device, the method comprising: obtaining a tth network status of a transmission network where the multimodal data is located at a tth time; determining a compression strategy j with the minimum total cost for the multimodal data under the tth network status;

[0005] Each modal data in the multimodal data is compressed using compression strategy j; wherein the total cost for compression strategy m is determined based on the instantaneous cost at the tth moment when the multimodal data adopts the compression strategy m under the tth network condition and the impact cost at the tth moment; the instantaneous cost at the tth moment represents the impact on the quality and / or transmission speed of the multimodal data after compression; the impact cost at the tth moment represents the sum of the minimum total costs at the t-1th moment when various compression strategies are adopted under various network conditions at the t-1th moment, with the corresponding network state transition probability as the weight; the corresponding network state transition probability is the probability that any network state at the t-1th moment is converted to the tth network state after the compression strategy m is adopted.

[0006] In a possible implementation, the instantaneous cost at the tth moment is determined in the following manner, including: obtaining the compression rate of the compression strategy m; determining the amount of data after compression of each modal data according to the data amount of each modal data in the multimodal data and the compression rate of the compression strategy m; determining the transmission time required for transmission under the tth network condition according to the sum of the data amounts corresponding to the compressed data amounts of each modal data; determining the instantaneous cost at the tth moment according to the transmission time and the quality loss corresponding to the compression strategy m.

[0007] In a possible implementation, the probability of any network state transition is obtained based on statistical data of a network state transitioning to the qth network state at the next moment after the nth compression strategy is adopted in the pth network state at the previous moment during a historical period.

[0008] In one possible implementation, various network conditions include low bandwidth, medium bandwidth and high bandwidth; various compression strategies include low compression strategy, medium compression strategy and high compression strategy; various modal data include video, audio and text; the compression rates of different modal data under the same compression strategy may be different, and the quality losses of different modal data under the same compression strategy are the same.

[0009] In a possible implementation, the impact cost at the tth moment is determined by the following formula, including:

[0010] ∑ k (P[t|k,m]·V[t-1][k])

[0011] Wherein, P[t|k,m] represents the network state transition probability of converting to the tth network state after the compression strategy m is adopted under the kth network state at the t-1th time; the kth network state is any one of various network states; V[t-1][k] represents the smallest total cost of compressing the multimodal data using various compression strategies under the kth network state at the t-1th time.

[0012] In a possible implementation, the compression strategy j with the minimum total cost for the multimodal data under the t-th network condition is determined by the following formula, including:

[0013] V[t][i]=(C[t][i][j]+∑ k P[t|k,j]·V[t-1][k])

[0014] Among them, C[t][i][j] represents the instantaneous cost of compressing the multimodal data using compression strategy j under the i-th network condition at the t-th time; the i-th network condition is the t-th network condition; V[t][i] represents the smallest total cost among the total costs of compressing the multimodal data using various compression strategies under the i-th network condition at the t-th time.

[0015] In a second aspect, an embodiment of the present application provides a multimodal data compression device, the device comprising: an acquisition module for acquiring the tth network status of the transmission network where the multimodal data is located at the tth time; a determination module for determining the compression strategy j with the minimum total cost for the multimodal data under the tth network status; each modal data in the multimodal data is compressed using the compression strategy j; wherein the total cost for the compression strategy m is determined based on the instantaneous cost at the tth time when the multimodal data adopts the compression strategy m under the tth network status and the impact cost at the tth time; the instantaneous cost at the tth time represents the impact on the quality and / or transmission speed of the multimodal data after compression; the impact cost at the tth time represents the sum of the minimum total costs at the t-1th time when various compression strategies are adopted under various network conditions at the t-1th time, with the corresponding network state transition probability as the weight; the corresponding network state transition probability is the probability that any network state at the t-1th time is converted to the tth network state after adopting the compression strategy m.

[0016] In a possible implementation, the acquisition module is also used to obtain the compression rate of the compression strategy m; the determination module is specifically used to determine the amount of data after compression of each modal data according to the data amount of each modal data in the multimodal data and the compression rate of the compression strategy m; determine the transmission time required for transmission under the tth network condition according to the sum of the data amounts corresponding to the compressed data amounts of each modal data; determine the instantaneous cost at the tth moment according to the transmission time and the quality loss corresponding to the compression strategy m.

[0017] In a possible implementation, the device further includes a statistical module for counting, within a historical period, whether the network status is converted to the qth network status at the next moment after the nth compression strategy is adopted when the network status is the pth network status at the previous moment.

[0018] In one possible implementation, various network conditions include low bandwidth, medium bandwidth and high bandwidth; various compression strategies include low compression strategy, medium compression strategy and high compression strategy; various modal data include video, audio and text; the compression rates of different modal data under the same compression strategy may be different, and the quality losses of different modal data under the same compression strategy are the same.

[0019] In a possible implementation, the determining module determines the impact cost at the tth moment by using the following formula, including:

[0020] ∑ k (P[t|k,m]·V[t-1][k])

[0021] Wherein, P[t|k,m] represents the network state transition probability of converting to the tth network state after the compression strategy m is adopted under the kth network state at the t-1th time; the kth network state is any one of various network states; V[t-1][k] represents the smallest total cost of compressing the multimodal data using various compression strategies under the kth network state at the t-1th time.

[0022] In a possible implementation, the determination module determines the compression strategy j with the minimum total cost for the multimodal data under the t-th network condition by the following formula, including:

[0023] V[t][i]=C[t][i][j]+∑ k (P[t|k,j]·V[t-1][k])

[0024] Among them, C[t][i][j] represents the instantaneous cost of compressing the multimodal data using compression strategy j under the i-th network condition at the t-th time; the i-th network condition is the t-th network condition; V[t][i] represents the smallest total cost among the total costs of compressing the multimodal data using various compression strategies under the i-th network condition at the t-th time.

[0025] In a third aspect, an embodiment of the present application further provides a multimodal data compression device, comprising a memory and a processor, wherein the memory is used to store computer programs or instructions; the processor is used to call the computer programs or instructions stored in the memory to execute a method as in any possible implementation of the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When a computer reads and executes the instructions, the computer executes a method in any possible implementation of the first aspect.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein instructions are stored in the computer program product. When a computer reads and executes the instructions, the computer executes the method in any possible implementation of the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A schematic diagram of a transmission scenario of compressed multimodal data in a metaverse scenario provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of a flow chart corresponding to a multimodal data compression method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the internal modules of a multimodal data compression device 3000 provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of a multimodal data compression device 4000 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0034] With the development of science and technology, many scenarios require users to receive various types of data, including text, images, audio, and video. For example, when users wear virtual reality (VR) glasses to watch VR videos, they need to receive audio, video, and text at the same time. For example, when holding a meeting through conference software, users also need to receive or send text, images, audio, and video at the same time. Multiple types of data can be called multimodal data. In real life, due to network or other factors, files containing various data types often need to be compressed and transmitted to provide a more comprehensive user experience.

[0035] The existing data compression method for multimodal data usually compresses each type of data separately, and the compression of various types of data lacks coordination and consistency because different types of data may have different characteristics and requirements. In addition, the existing data compression cannot flexibly adjust the compression strategy according to changes in network conditions.

[0036] Based on this, an embodiment of the present application provides a multimodal data compression method for flexibly adjusting the compression strategy under dynamic changes in network status.

[0037] Figure 1 A schematic diagram of a transmission scenario of compressed multimodal data in a metaverse scenario provided by an embodiment of the present application. Figure 1It can be seen that in a virtual exhibition, the online exhibition server can display data such as pictures, videos, audios, and texts of the exhibition content. The online exhibition server can be a computer, server, or other intelligent device. The network conditions of each user connection can be different. The multimodal data compression device compresses the multimodal data based on the multimodal data compression method according to the network conditions of each user connection, thereby reducing the cost of multimodal data transmission and storage. The multimodal data compression device can be an online exhibition server, or it can be another device for compressing multimodal data connected to the online exhibition server. Figure 1 The device is not shown in the video. The compressed multimodal data is transmitted to each user's device, such as a VR helmet, through a network connection. Then the application on each user's device decompresses and renders the data, and displays the updated virtual exhibition content in real time, such as high-resolution images of artworks, spatial audio effects, etc.

[0038] The multimodal data compression method provided in the embodiment of the present application can be used not only for multimodal data compression in the metaverse scenario, but also in other scenarios, such as each user watching the host's live broadcast through his own smart device, and each user watching online videos through his own smart device. No more examples will be given here.

[0039] Figure 2 A schematic diagram of a flow chart corresponding to a multimodal data compression method provided in an embodiment of the present application, the flow chart can be executed by a multimodal data compression device, such as Figure 2 As shown, the process includes the following steps:

[0040] Step 201: The multimodal data compression apparatus obtains the tth network status of the transmission network where the multimodal data is located at the tth time.

[0041] Specifically, the network status is a dynamically changing indicator that affects the efficiency and quality of data transmission. The network status may include network bandwidth, network delay, network connection stability, and packet loss rate. Each moment has a corresponding network status. In the embodiment of the present application, the network status at the first moment is recorded as the first network status, and the network status at the tth moment is recorded as the tth network status.

[0042] Step 202: The multimodal data compression apparatus determines a compression strategy j with the minimum total cost for the multimodal data under the tth network condition.

[0043] Specifically, the embodiments of the present application include a variety of compression strategies. From the perspective of compression rate, the various compression strategies can be divided into low compression strategy, medium compression strategy and high compression strategy, wherein the high compression strategy reduces the data volume to a large extent, usually accompanied by a significant loss of data quality, which may lead to a large loss of information, such as a decrease in the quality of image, audio or video data; the medium compression strategy usually balances between the volume and quality of the data, using a relatively moderate compression rate to reduce storage or transmission overhead while maintaining a certain data quality, the data quality is still acceptable, and the user may not easily notice the quality decrease; the low compression strategy usually causes the data to be slightly compressed, mainly to reduce the storage volume without sacrificing too much quality or information integrity, and the quality is almost unchanged or slightly decreased. In the embodiments of the present application, the compression rate of the high compression strategy indicates that the ratio of the compressed file size to the original file size is below 0.5; the compression rate of the medium compression strategy indicates that the ratio of the compressed file size to the original file size is between 0.5 and 0.8; the compression rate of the low compression strategy indicates that the ratio of the compressed file size to the original file size is above 0.8. The compression rate of the high compression strategy is greater than that of the medium compression strategy, and the compression rate of the medium compression strategy is greater than that of the low compression strategy, but the present application does not specifically limit the specific values ​​of the compression rates of various compression strategies. In an embodiment of the present application, the compression rates of different modal data under the same compression strategy may be different. For example, various modal data include video, audio, and text. The compression rates of video, audio, and text under the high compression strategy are 0.2, 0.2, and 0.5, respectively; the compression rates under the medium compression strategy are 0.5, 0.5, and 0.7, respectively; and the compression rates under the low compression strategy are 0.8, 0.8, and 0.9, respectively. Optionally, in an embodiment of the present application, the quality loss of different modal data under the same compression strategy is the same. If 0 represents the minimum quality loss, the quality loss corresponding to the high compression strategy is 0.5, the quality loss corresponding to the medium compression strategy is 0.25, and the quality loss corresponding to the low compression strategy is 0.05.

[0044] Furthermore, the total costs of various compression strategies for multimodal data under the t-th network condition at the t-th time are not the same. In order to determine the compression strategy j with the minimum total cost for multimodal data under the t-th network condition, that is, to determine which compression strategy, a high compression strategy, a medium compression strategy or a low compression strategy, can minimize the total cost of compressing multimodal data under the t-th network condition at the t-th time, it is necessary to calculate the total cost of compressing multimodal data under each compression strategy. It should be noted that the embodiment of the present application uses the same compression strategy to compress each modal data in the multimodal data.

[0045] Specifically, the calculation of the total cost of compressing multimodal data using compression strategy m under the tth network condition is taken as an example to describe how to calculate the total cost.

[0046] The total cost for compression strategy m is determined based on the instantaneous cost of taking compression strategy m at the tth moment and the impact cost at the tth moment for multimodal data under the tth network condition.

[0047] (1) Calculate the immediate cost

[0048] The instantaneous cost at time t represents the impact on the quality and / or transmission speed of the multimodal data after compression.

[0049] Specifically, the network condition may include network bandwidth, and various network conditions include low bandwidth, medium bandwidth, and high bandwidth. For example, in the embodiment of the present application, a bandwidth between 0 and 50 Mbps is defined as low bandwidth; a bandwidth between 50 and 100 Mbps is defined as medium bandwidth; and a bandwidth above 100 Mbps is defined as high bandwidth. The embodiment of the present application does not limit the number of classifications of network bandwidth and the specific numerical standards of the classifications.

[0050] The compression ratio of compression strategy m is obtained, and the data volume of each modal data after compression is determined according to the data volume of each modal data in the multimodal data and the compression ratio of compression strategy m. For example, the data volume of each modal data in the multimodal data is 500MB for video, 50MB for audio, and 1MB for text, and compression strategy m is a medium compression strategy. Therefore, the data volume of the video data after compression is 500*0.5=250MB, the data volume of the audio data after compression is 50*0.5=25MB, and the data volume of the text data after compression is 1*0.7=0.7MB.

[0051] According to the sum of the data volume corresponding to the compressed data volume of each modal data, the transmission time required for transmission under the tth network condition is determined. That is, the sum of the data volume corresponding to the compressed data volume of each modal data is calculated to be 250+25+0.7=275.7MB. If the bandwidth of the tth network condition is 120MB, the transmission time is 275.7*8 / 120MB=18.38s.

[0052] Optionally, in addition to determining the transmission time required for transmission, the speed corresponding to the transmission under the tth network condition can also be determined based on the sum of the data volume corresponding to the data volume after compression of each modal data. Preferably, the instant cost is calculated based on the transmission time as a consideration in the embodiment of the present application.

[0053] Furthermore, the instantaneous cost at the tth moment is determined according to the transmission time and the quality loss corresponding to the compression strategy m. According to the above description of the compression strategy, the quality loss of the compression strategy is 0.25. According to the sum of the transmission time and the quality loss, the instantaneous cost of adopting the compression strategy m to compress the above multimodal data under the tth network condition is determined to be 18.38+0.25=18.63. It can be seen that the instantaneous cost represents the cost of the transmission time and quality loss required to adopt the compression strategy under the current network state.

[0054] Optionally, the instantaneous cost may be expressed only according to the transmission time of the compressed multimodal data, or only according to the quality loss of the compressed multimodal data. The instantaneous cost may also be calculated based on the transmission time and the quality loss by other calculation methods, which will not be described in detail in the embodiments of the present application.

[0055] (2) Calculate impact costs

[0056] The impact cost at the t-th moment represents the sum of the minimum total costs at the t-1th moment when various compression strategies are adopted under various network conditions at the t-1th moment, with the corresponding network state transition probabilities as weights.

[0057] Specifically, the corresponding network state transition probability is the probability that any network state at the t-1th moment is converted to the tth network state after the compression strategy m is adopted. Optionally, any network state transition probability is obtained based on the statistical data of the conversion to the qth network state at the next moment after the nth compression strategy is adopted when the previous moment is the pth network state in the historical period. For example, the pth network state is medium bandwidth, the nth compression strategy is a low compression strategy, and the qth network state is high bandwidth. According to the conversion of network conditions in the past month, the proportion of the medium bandwidth network state converted to high bandwidth at the next moment after compression with a low compression strategy is calculated.

[0058] In one possible approach, the impact cost at time t is determined by the following formula, including:

[0059] ∑ k (P[t|k,m]·V[t-1][k])

[0060] Among them, P[t|k,m] represents the network state transition probability that the network state at the (t - 1)-th moment is converted to the t-th network state after adopting the compression strategy m under the k-th network condition, where the k-th network condition is any one of various network conditions, and V[t - 1][k] represents the minimum total cost among the total costs of compressing multimodal data by adopting various compression strategies under the k-th network condition at the (t - 1)-th moment. V[t - 1][k] is also determined according to the immediate cost and the impact cost. Thus, it can be seen that the impact cost represents the cost of the impact on the current moment by adopting various compression strategies under various network conditions at the previous moment.

[0061] Therefore, at the t-th moment under the t-th network condition, the total cost of compressing multimodal data by adopting the compression strategy m can be determined by the sum of the immediate cost and the impact cost of adopting the compression strategy m. Further, the compression strategy j with the minimum total cost for multimodal data under the t-th network condition is determined by the following formula, including:

[0062] V[t][i] = C[t][i][j] + ∑ k (P[t|k,j]·V[t - 1][k])

[0063] = min{C[t][i][high] + ∑ k (P[t|k,high]·V[t - 1][k]), C[t][i][medium] +

[0064] ∑ k (P[t|k,medium]·V[t - 1][k]), C[t][i][low] + ∑ k (P[t|k,low]·V[t - 1][k])}

[0065] Among them, C[t][i][j] represents the immediate cost of compressing multimodal data by adopting the compression strategy j under the i-th network condition at the t-th moment, where the i-th network condition is the t-th network condition, min represents taking the minimum value of the total cost under various compression strategies, and j is the compression strategy corresponding to the minimum value of the total cost under various compression strategies.

[0066] It can be understood that V[t - 1][k] = V[t - 1][i ′ = C[t - 1][i ′ [j] + ∑ k (P[t - 1|k,j]·V[t - 2][k]) = min{C[t - 1][i ′ [high] + ∑ k (P[t - 1|k,high]·V[t - 2][k]), C[t - 1][i ′ [medium] + ∑ k(P[t - 1|k, medium]·V[t - 2][k]), C[t - 1][i ′ [low] +

[0067] ∑ k (P[t - 1|k, low]·V[t - 2][k])}

[0068] In the formula, i ′ represents the network state at the (t - 1)-th moment. In this way, by gradually recursing, for each time point t, according to the intelligently adjusted decision, the value of V[t][i] can be obtained. Then, according to the network state at time point t, the compression strategy j that minimizes V[t][i] is selected, which can ensure that the compressed data can maintain high quality after decompression and meet the requirements of the metaverse application for data accuracy. It should be noted that an initial condition needs to be preset. According to the experience of the staff, it can be set that at t = 0, for each network state i, the minimum total cost of adopting each compression strategy is V[0][i] = 1.

[0069] Figure 3 FIG. is a schematic diagram of the internal modules of a multimodal data compression device 3000 provided by an embodiment of the present application. As Figure 3 shown, the device may include: an acquisition module 301, a determination module 302, and a statistics module 303. Optionally, it further includes a storage module for storing computer instructions or programs. The determination module 302 may call the computer instructions or programs in the storage module.

[0070] Among them, the acquisition module 301 is used to acquire the t-th network condition of the transmission network where the multimodal data is located at the t-th moment; the determination module 302 is used to determine the compression strategy j with the minimum total cost for the multimodal data under the t-th network condition; and each modality data in the multimodal data is compressed using the compression strategy j. Among them, the total cost for the compression strategy m is determined according to the immediate cost at the t-th moment and the impact cost at the t-th moment of the multimodal data adopting the compression strategy m under the t-th network condition. The immediate cost at the t-th moment characterizes the quality impact and / or transmission speed impact on the multimodal data after compression; the impact cost at the t-th moment characterizes the sum of the minimum total costs at the (t - 1)-th moment with the corresponding network state transition probabilities as weights when adopting various compression strategies under various network conditions at the (t - 1)-th moment; the corresponding network state transition probability is the probability that any network condition at the (t - 1)-th moment is converted to the t-th network condition after adopting the compression strategy m.

[0071] In a possible implementation, the acquisition module 301 is also used to obtain the compression rate of the compression strategy m; the determination module 302 is specifically used to determine the amount of data after compression of each modal data according to the data amount of each modal data in the multimodal data and the compression rate of the compression strategy m; determine the transmission time required for transmission under the tth network condition according to the sum of the data amounts corresponding to the compressed data amounts of each modal data; determine the instantaneous cost at the tth moment according to the transmission time and the quality loss corresponding to the compression strategy m.

[0072] In a possible implementation, the device further includes a statistical module 303 for counting, within a historical period, whether the network status is converted to the qth network status at the next moment after the nth compression strategy is adopted when the network status is the pth network status at the previous moment.

[0073] In one possible implementation, various network conditions include low bandwidth, medium bandwidth and high bandwidth; various compression strategies include low compression strategy, medium compression strategy and high compression strategy; various modal data include video, audio and text; the compression rates of different modal data under the same compression strategy may be different, and the quality losses of different modal data under the same compression strategy are the same.

[0074] In a possible implementation, the determining module 302 determines the impact cost at the tth moment by using the following formula, including:

[0075] ∑ k (P[t|k,m]·V[t-1][k])

[0076] Wherein, P[t|k,m] represents the network state transition probability of converting to the tth network state after the compression strategy m is adopted under the kth network state at the t-1th time; the kth network state is any one of various network states; V[t-1][k] represents the smallest total cost of compressing the multimodal data using various compression strategies under the kth network state at the t-1th time.

[0077] In a possible implementation, the determination module 302 determines the compression strategy j with the minimum total cost for the multimodal data under the t-th network condition by using the following formula, including:

[0078] V[t][i]=C[t][i][j]+∑ k (P[t|k,j]·V[t-1][k])

[0079] Among them, C[t][i][j] represents the instantaneous cost of compressing the multimodal data using compression strategy j under the i-th network condition at the t-th time; the i-th network condition is the t-th network condition; V[t][i] represents the smallest total cost among the total costs of compressing the multimodal data using various compression strategies under the i-th network condition at the t-th time.

[0080] Figure 4 Schematic diagram of a multi-modal data compression device 4000 provided in an embodiment of the present application. Figure 4 As shown, it includes at least one processor 401 and a memory 402 connected to the at least one processor 401. The specific connection medium between the processor 401 and the memory 402 is not limited in the embodiment of the present application. Figure 4 For example, the processor 401 and the memory 402 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0081] In the embodiment of the present invention, the memory 402 stores instructions that can be executed by at least one processor 401, and the at least one processor 401 can implement the steps of the above-mentioned multimodal data compression method by executing the instructions stored in the memory 402.

[0082] The processor 401 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, and perform resource settings by running or executing instructions stored in the memory 402 and calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing units, and the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0083] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0084] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal data compression method, characterized in that: The method comprises: Obtaining the t-th network status of the transmission network where the multimodal data is located at the t-th time; Determine a compression strategy j with the minimum total cost for multimodal data under the t-th network condition; Each modal data in the multimodal data is compressed using compression strategy j; wherein the total cost for compression strategy m is determined based on the instantaneous cost at the tth moment when the multimodal data adopts the compression strategy m under the tth network condition and the impact cost at the tth moment; the instantaneous cost at the tth moment represents the impact on the quality and / or transmission speed of the multimodal data after compression; the impact cost at the tth moment represents the sum of the minimum total costs at the t-1th moment when various compression strategies are adopted under various network conditions at the t-1th moment, with the corresponding network state transition probability as the weight; the corresponding network state transition probability is the probability that any network state at the t-1th moment is converted to the tth network state after the compression strategy m is adopted.

2. The method according to claim 1, characterized in that The instantaneous cost at the tth moment is determined by the following method, including: Obtaining the compression ratio of the compression strategy m; Determining the amount of data after compression of each modal data according to the amount of data of each modal data in the multimodal data and the compression rate of the compression strategy m; Determine the transmission time required for transmission under the t-th network condition according to the sum of the data amounts corresponding to the data amounts after compression of each modal data; The instantaneous cost at the tth moment is determined according to the transmission time and the quality loss corresponding to the compression strategy m.

3. The method according to claim 1, characterized in that The probability of any network state transition is obtained based on the statistical data of the network state being converted to the qth network state at the next moment after the nth compression strategy is adopted when the network state is the pth network state at the previous moment in the historical period.

4. The method according to claim 3, characterized in that The method comprises: Various network conditions include low bandwidth, medium bandwidth, and high bandwidth; Various compression strategies include low compression strategy, medium compression strategy and high compression strategy; Various modal data include video, audio, and text; The compression rates of different modal data under the same compression strategy may be different, and the quality losses of different modal data under the same compression strategy are the same.

5. The method according to any one of claims 1 to 4, characterized in that: The impact cost at the time t is determined by the following formula, including: ∑ k (P[t|k,m]·V[t-1][k]) Wherein, P[t|k,m] represents the network state transition probability of converting to the t-th network state after adopting compression strategy m under the k-th network state at time t-1; the k-th network state is any one of various network states; V[t-1][k] represents the minimum total cost among the total costs of compressing the multimodal data using various compression strategies under the kth network condition at the t-1th time.

6. The method according to claim 5, characterized in that The compression strategy j with the minimum total cost for multimodal data under the t-th network condition is determined by the following formula, including: V[t][i]=C[t][i][j]+∑ k (P[t|k,j]·V[t-1][k]) Wherein, C[t][i][j] represents the instantaneous cost of compressing the multimodal data using compression strategy j under the i-th network condition at the t-th time; the i-th network condition is the t-th network condition; V[t][i] represents the minimum total cost among the total costs of compressing the multimodal data using various compression strategies under the i-th network condition at the t-th time.

7. A multimodal data compression device, characterized in that: include: An acquisition module, used to acquire the tth network status of the transmission network where the multimodal data is located at the tth time; A determination module is used to determine the compression strategy j with the minimum total cost for multimodal data under the t-th network condition; each modal data in the multimodal data is compressed using the compression strategy j; wherein the total cost for the compression strategy m is determined based on the instantaneous cost at the t-th moment when the multimodal data adopts the compression strategy m under the t-th network condition and the impact cost at the t-th moment; the instantaneous cost at the t-th moment represents the impact on the quality and / or transmission speed of the multimodal data after compression; the impact cost at the t-th moment represents the sum of the minimum total costs at the t-1th moment when various compression strategies are adopted under various network conditions at the t-1th moment with the corresponding network state transition probability as the weight; the corresponding network state transition probability is the probability that any network state at the t-1th moment is converted to the t-th network state after adopting the compression strategy m.

8. A multimodal data compression device, characterized in that: include: Memory, used to store computer programs or instructions; A processor, configured to call a computer program or instruction stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when a computer reads and executes the instructions, the computer executes the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product stores instructions, and when a computer reads and executes the instructions, the computer executes the method according to any one of claims 1 to 6.

Images