Image processing method, device and system, storage medium and program product

By splitting the images in the video stream and encoding them by combining hard coding and soft coding, the delay and lag problems between the VR server and the client are solved, and more efficient video streaming is achieved. Improved user experience.

CN119011818AActive Publication Date: 2024-11-22MOORE THREADS TECHNOLOGY (SHANGHAI) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411098139.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-22
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

In the case of high image resolution and low working efficiency of encoding and decoding devices, delays and lags often occur in video streaming between the virtual reality (VR) server and the client, reducing the user's user experience.

Method used

By splitting the images in the video stream into multiple sub-images, and encoding sub-images with different resolutions is combined by combining hard-coded and soft-coded, the hard-coded paths of the server device are used for parallel encoding to reduce delay time and lag phenomenon.

Benefits of technology

The video stream is encoded and coded by combining soft codec and hard codec, making full use of encoding and decoding, reducing delays and lags, and improving user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011818A_ABST
    Figure CN119011818A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image processing method, device and system, a storage medium and a program product, and the method comprises the steps: obtaining a to-be-coded image, and obtaining a splitting parameter and a coding parameter, the splitting parameter is used for splitting the to-be-coded image into a plurality of sub-images, and the coding parameter comprises coding types corresponding to the plurality of sub-images, the coding type comprises hard coding and / or soft coding, splitting the to-be-coded image according to the splitting parameter to obtain a plurality of first images, coding the plurality of first images according to the plurality of coding types to obtain a plurality of to-be-decoded images, and sending the plurality of to-be-decoded images to the client device, the plurality of first images and the plurality of to-be-decoded images are in one-to-one correspondence. According to the image processing method provided by the embodiment of the invention, the delay and lagging phenomena can be reduced, and the use experience of a user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular, to an image processing method, device, system, storage medium and program product. Background Art

[0002] In order to efficiently transmit and store video data on different devices and platforms, video streams are usually encoded or decoded.

[0003] In the related art, between a virtual reality (VR) server and a client, the server usually hard codes or soft codes the image to be processed, and sends the encoded video stream to the client, and then the client hard-decodes or soft-decodes the encoded video stream, and then displays the decoded video stream.

[0004] However, in the process of implementing this application, the inventors found that there are at least the following problems in the prior art: in the above-mentioned method, when the image resolution is high and the efficiency of the encoding and decoding equipment is low, delays and freezes often occur, which reduces the user experience. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, device, system, storage medium and program product to reduce delay and freeze and improve user experience.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a server device, including:

[0007] Acquire an image to be encoded and acquire a splitting parameter and an encoding parameter; the splitting parameter is used to split the image to be encoded into multiple sub-images; the encoding parameter includes encoding types corresponding to the multiple sub-images respectively; the encoding type includes hard coding and / or soft coding;

[0008] Splitting the to-be-encoded image according to the splitting parameter to obtain a plurality of first images;

[0009] According to the multiple encoding types, the multiple first images are encoded to obtain multiple images to be decoded, so as to send the multiple images to be decoded to a client device; the multiple first images and the multiple images to be decoded are in a one-to-one correspondence.

[0010] In a possible design, the splitting parameter includes at least one set of resolutions and a preset number corresponding to each set of resolutions; each set of resolutions has the same resolution value, and different sets of resolutions have different resolution values; and splitting the image to be encoded according to the splitting parameter to obtain a plurality of first images includes:

[0011] According to the at least one set of resolutions and the preset numbers respectively corresponding to the at least one set of resolutions, the image to be encoded is split to obtain a plurality of first images; the plurality of first images include a corresponding preset number of images at each of the resolutions.

[0012] In a possible design, encoding the multiple first images according to the multiple encoding types to obtain multiple images to be decoded includes:

[0013] If there are images to be hard-coded whose corresponding encoding type is hard coding among the multiple first images, and there are images to be soft-coded whose corresponding encoding type is soft coding, hard coding of the images to be hard-coded and soft coding of the images to be soft-coded are performed in parallel to obtain multiple images to be decoded.

[0014] In a possible design, the encoding parameters further include the number of hard encoding paths supported by the server device; and the hard encoding of the image to be hard-coded and the soft encoding of the image to be soft-coded are performed in parallel to obtain a plurality of images to be decoded, including:

[0015] If the number of hard-coding paths is multiple, when hard-coding the image to be hard-coded, the image to be hard-coded is divided into multiple paths for parallel hard coding according to the number of hard-coding paths.

[0016] In one possible design, encoding the multiple first images according to the multiple encoding types includes:

[0017] Get the encoding format;

[0018] The plurality of first images are encoded according to the plurality of encoding types and the encoding format.

[0019] In a possible design, before obtaining the image to be encoded and obtaining the splitting parameters and the encoding parameters, the process further includes:

[0020] Acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters;

[0021] For each image group in the image combination information, a plurality of decoding times corresponding to the image group is obtained; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations;

[0022] For each image subassembly group in the image combination information, a plurality of encoding times corresponding to the image group is obtained; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations;

[0023] The splitting parameter and the encoding parameter are determined according to the multiple decoding times and the multiple encoding times.

[0024] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to a client device, and the method includes:

[0025] Acquire multiple images to be decoded sent by the server device; the multiple images to be decoded are obtained by the server device based on splitting parameters and encoding parameters; the splitting parameters are used to split the image to be encoded into multiple sub-images; the encoding parameters include encoding types corresponding to the multiple sub-images respectively;

[0026] Obtaining decoding parameters and splicing parameters; the decoding parameters include decoding types corresponding to the multiple sub-images respectively; the splicing parameters are used to characterize the relative position relationship of the multiple sub-images;

[0027] Decoding the multiple images to be decoded according to the multiple decoding types to obtain multiple second images;

[0028] The plurality of second images are stitched together according to the stitching parameters to obtain a target image.

[0029] In a possible design, the decoding type includes hard decoding or soft decoding; and decoding the multiple images to be decoded according to the multiple decoding types to obtain multiple second images includes:

[0030] If there are images to be hard-decoded whose corresponding decoding type is hard decoding among the multiple images to be decoded, and there are images to be soft-decoded whose corresponding decoding type is soft decoding, hard decoding of the images to be hard-decoded and soft decoding of the images to be soft-decoded are performed in parallel to obtain multiple second images.

[0031] In a possible design, the decoding parameter further includes the number of hard decoding paths supported by the client device; and the hard decoding of the image to be hard-decoded and the soft decoding of the image to be soft-decoded are performed in parallel to obtain a plurality of second images, including:

[0032] If the number of hard decoding paths is multiple, when hard decoding is performed on the image to be hard-decoded, the image to be hard-decoded is divided into multiple paths for parallel hard decoding according to the number of hard decoding paths.

[0033] In a possible design, the stitching parameters include position information of each sub-image on the image to be encoded; stitching the multiple second images according to the stitching parameters to obtain a target image includes:

[0034] The multiple second images are spliced ​​according to the position information of the sub-images respectively corresponding to the multiple second images on the image to be encoded to obtain a target image.

[0035] In a possible design, before obtaining the decoding parameters and the splicing parameters, the process further includes:

[0036] Acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters;

[0037] For each image group in the image combination information, a plurality of decoding times corresponding to the image group is obtained; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations;

[0038] For each image subassembly group in the image combination information, a plurality of encoding times corresponding to the image group is obtained; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations;

[0039] The splicing parameter and the decoding parameter are determined according to the multiple decoding time consumptions and the multiple encoding time consumptions.

[0040] In a third aspect, an embodiment of the present application provides a method for determining image encoding and decoding parameters, including:

[0041] Acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters;

[0042] For each image group in the image combination information, a plurality of decoding times corresponding to the image group is obtained; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations;

[0043] For each image subassembly group in the image combination information, a plurality of encoding times corresponding to the image group is obtained; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations;

[0044] According to the multiple decoding times and the multiple encoding times, target splitting parameters, target splicing parameters, target decoding parameters and target encoding parameters are determined; so that the server device splits and encodes the image to be encoded according to the target splitting parameters and the target decoding parameters to obtain the image to be decoded, and the client device decodes and splices the image to be decoded according to the target decoding parameters and the target splicing parameters to obtain the target image.

[0045] In a possible design, determining a target splitting parameter, a target splicing parameter, a target decoding parameter, and a target encoding parameter according to the plurality of decoding times and the plurality of encoding times includes:

[0046] According to the plurality of decoding times and the plurality of encoding times, determine the sum of the decoding time and the encoding time of each image group in each encoding mode and in each decoding mode, and determine the image group corresponding to the minimum sum as the target image group, the decoding mode corresponding to the minimum sum as the target decoding mode, and the encoding mode corresponding to the minimum sum as the target encoding mode;

[0047] The target splitting parameters and the target splicing parameters are determined according to the target image grouping, the target decoding parameters are determined according to the target decoding mode, and the target encoding parameters are determined according to the target encoding mode.

[0048] In a possible design, applied to a server device, obtaining, for each image group in the image combination information, a plurality of decoding times corresponding to the image group includes:

[0049] Sending the image combination information to the client device, so that the client device decodes the images in the image group based on the multiple decoding modes for each image group in the image combination information, and obtains multiple decoding times corresponding to the image group;

[0050] Receive multiple decoding times corresponding to the multiple image groups sent by the client device.

[0051] In a possible design, applied to a server device, obtaining, for each image group in the image combination information, a plurality of decoding times corresponding to the image group includes:

[0052] Sending the image combination information to the server device, so that the server device encodes the images in the image group based on the multiple encoding modes for each image group in the image combination information, and obtains multiple encoding times corresponding to the image group;

[0053] Receiving multiple encoding times corresponding to the multiple image groups sent by the server device.

[0054] In a fourth aspect, an embodiment of the present application provides a server device, including:

[0055] A first acquisition module is used to acquire an image to be encoded and to acquire a splitting parameter and an encoding parameter; the splitting parameter is used to split the image to be encoded into multiple sub-images; the encoding parameter includes encoding types corresponding to the multiple sub-images respectively; the encoding type includes hard coding and / or soft coding;

[0056] A splitting module, used for splitting the image to be encoded according to the splitting parameter to obtain a plurality of first images;

[0057] The encoding module is used to encode the multiple first images according to the multiple encoding types to obtain multiple images to be decoded, so as to send the multiple images to be decoded to the client device; the multiple first images and the multiple images to be decoded are in a one-to-one correspondence.

[0058] In a fifth aspect, an embodiment of the present application provides a client device, including:

[0059] A receiving module, used for acquiring a plurality of images to be decoded sent by a server device; the plurality of images to be decoded are obtained by the server device based on a splitting parameter and a coding parameter; the splitting parameter is used for splitting the image to be encoded into a plurality of sub-images; the coding parameter includes coding types corresponding to the plurality of sub-images respectively;

[0060] A second acquisition module is used to acquire decoding parameters and splicing parameters; the decoding parameters include decoding types corresponding to the multiple sub-images respectively; the splicing parameters are used to characterize the relative position relationship of the multiple sub-images;

[0061] A decoding module, configured to decode the plurality of images to be decoded according to the plurality of decoding types to obtain a plurality of second images;

[0062] The stitching module is used to stitch the multiple second images according to the stitching parameters to obtain a target image.

[0063] In a sixth aspect, an embodiment of the present application provides an image processing parameter determination device, including:

[0064] A third acquisition module is used to acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters;

[0065] a fourth acquisition module, configured to acquire, for each image group in the image combination information, a plurality of decoding times corresponding to the image group; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes decoding types respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations;

[0066] a fifth acquisition module, configured to acquire, for each image sub-group in the image combination information, a plurality of encoding times corresponding to the image group; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations;

[0067] A determination module is used to determine a target splitting parameter, a target splicing parameter, a target decoding parameter and a target encoding parameter according to the multiple decoding time consumptions and the multiple encoding time consumptions; so that the server device splits and encodes the image to be encoded according to the target splitting parameter and the target decoding parameter to obtain the image to be decoded, and the client device decodes and splices the image to be decoded according to the target decoding parameter and the target splicing parameter to obtain the target image.

[0068] In a seventh aspect, an embodiment of the present application provides a server device, including: at least one processor and a memory;

[0069] The memory stores computer-executable instructions;

[0070] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect and various possible designs of the first aspect.

[0071] In an eighth aspect, an embodiment of the present application provides a client device, including: at least one processor and a memory;

[0072] The memory stores computer-executable instructions;

[0073] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the second aspect and various possible designs of the second aspect.

[0074] In a ninth aspect, an embodiment of the present application provides an image processing system, comprising: a server device as described in the seventh aspect and a client device as described in the eighth aspect.

[0075] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method described in the first, second, and third aspects and various possible designs of the first, second, and third aspects is implemented.

[0076] In the eleventh aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the first, second, and third aspects and various possible designs of the first, second, and third aspects.

[0077] The image processing method, device, system, storage medium and program product provided in the present embodiment include obtaining an image to be encoded and obtaining a splitting parameter and a coding parameter, wherein the splitting parameter is used to split the image to be encoded into multiple sub-images, and the coding parameter includes the coding types corresponding to the multiple sub-images, and the coding type includes hard coding and / or soft coding. According to the splitting parameter, the image to be encoded is split to obtain multiple first images, and according to the multiple coding types, the multiple first images are encoded to obtain multiple images to be decoded, so as to send the multiple images to be decoded to the client device, and the multiple first images and the multiple images to be decoded are in a one-to-one correspondence. The image processing method provided in the embodiment of the present application, by splitting the images in the video stream, and hard coding is used for some of the split images, and soft coding is used for the other part of the images, the encoding of the video stream can be completed by combining soft coding and hard coding, so as to fully utilize the resources of the encoding device, reduce delays and freezes, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0079] Figure 1 A schematic diagram of an image processing method and an application scenario of the image processing method provided in an embodiment of the present application;

[0080] Figure 2 A schematic diagram of a flow chart of a method for determining image processing parameters provided in an embodiment of the present application;

[0081] Figure 3 A schematic diagram of an application scenario of the method for determining image processing parameters provided in an embodiment of the present application;

[0082] Figure 4 A schematic diagram of image splitting provided in an embodiment of the present application;

[0083] Figure 5 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0084] Figure 6 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0085] Figure 7 A schematic diagram of the structure of a server device provided in an embodiment of the present application;

[0086] Figure 8 A schematic diagram of the structure of a client device provided in an embodiment of the present application;

[0087] Fig. 9 A schematic diagram of the structure of an image processing parameter determination device provided in an embodiment of the present application;

[0088] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0090] It should be noted that the image processing method, device, system, storage medium and program product provided by the present application can be used in the field of image processing, and can also be used in any field except the field of image processing. The application field of the image processing method, device, system, storage medium and program product provided by the present application is not limited.

[0091] In order to solve the above-mentioned technical problems of delay and stuttering, the inventors of the present application have discovered that the images in the video stream can be split, and some of the split images can be hard-coded, while the other part of the images can be soft-coded. Then, on the client, some of the encoded images can be hard-decoded, while the other part of the images can be soft-decoded. In this way, the encoding and decoding of the video stream can be completed by combining soft encoding and decoding with hard encoding and decoding, so as to make full use of the resources of the encoding and decoding equipment, reduce delays and stuttering, and enhance the user experience.

[0092] Figure 1 Schematic diagram of the image processing method and application scenario of the image processing method provided in the embodiment of the present application. Figure 1 As shown, the server device 101 and the client device 102 are connected via a network. The server device 101 can be a computer, a tablet computer, a server, a cloud server, etc., and the client device can be a VR device, a tablet computer, a phone watch, a mobile phone, a computer, etc.

[0093] In the specific implementation process, the server device 101 obtains the image to be encoded and obtains the splitting parameters and encoding parameters, the splitting parameters are used to split the image to be encoded into multiple sub-images; the encoding parameters include the encoding types corresponding to the multiple sub-images respectively, the encoding types include hard coding and / or soft coding, according to the splitting parameters, the image to be encoded is split to obtain multiple first images, according to the multiple encoding types, the multiple first images are encoded to obtain multiple images to be decoded, so as to send the multiple images to be decoded to the client device 102, and the multiple first images and the multiple images to be decoded are in a one-to-one correspondence. The client device 102 receives a plurality of images to be decoded sent by the server device 101, wherein the plurality of images to be decoded are obtained based on splitting parameters and encoding parameters, wherein the splitting parameters are used to split the image to be encoded into a plurality of sub-images, wherein the encoding parameters include the encoding types corresponding to the plurality of sub-images, respectively, and obtains decoding parameters and splicing parameters, wherein the decoding parameters include the decoding types corresponding to the plurality of sub-images, respectively, and the splicing parameters are used to characterize the relative positional relationship of the plurality of sub-images, and the plurality of images to be decoded are decoded according to the plurality of decoding types to obtain a plurality of second images, and the plurality of second images are spliced ​​according to the splicing parameters to obtain a target image. The processing method and decoding method provided in the embodiment of the present application split the images in the video stream, and hard-code the part of the images obtained by the splitting, while soft-code the other part of the images, and then hard-decode the part of the images in the encoded images at the client, while soft-decoding the other part of the images, thereby being able to complete the encoding and decoding of the video stream by combining soft encoding and hard encoding and decoding, so as to make full use of the encoding and decoding equipment resources, reduce delay and freeze phenomenon, and improve the user experience.

[0094] It should be noted that Figure 1 The scenario diagram shown is only an example. The encoding, decoding methods and scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0095] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0096] Figure 2 Schematic diagram of the process of determining the image processing parameters provided in the embodiment of the present application. Figure 2 As shown, the method includes:

[0097] 201. Obtain image combination information; the image combination information includes multiple image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters.

[0098] The execution subject of this embodiment may be any electronic device with data processing capability, such as a server device or a client device, or some steps may be executed by the server device and some steps may be executed by the client device.

[0099] The encoding parameters, decoding parameters, splicing parameters and splitting parameters determined by the image processing parameter determination method provided by this embodiment can be applied to the image processing methods provided by the following embodiments, for example: Figure 5 The embodiments shown and Figure 6 The embodiment shown.

[0100] In practical applications, such as Figure 3 As shown, the image combination information can be stored in the server device. When it is necessary to determine the image encoding and decoding parameters, that is, when self-test is required, the client can send a request to the server to request to establish a connection. After the server responds to the request, the connection between the client and the server is completed. After the connection is established, the server device can pass the pre-stored image combination information as a parameter to the client device. The image combination information includes multiple image groups, and each image group may include a coding format, the number of images, the resolution of each image, and the encoded image.

[0101] Different image groups are obtained by splitting the original image in different forms. For example, based on the total resolution of the image rendered by cloud VR binoculars, the image can be split into different resolution combinations, and the sum of the image resolutions of the split sub-images is equal to the total resolution of the original image. For example: the total image resolution is 4096x2048, then after two splitting methods, image group A and image group B can be obtained, where image group A includes 2 2048x2048, image group B includes 4 1024x512, and group C includes 2 1024x1024 plus 1 2048x2048.

[0102] 202. For each image group in the image combination information, obtain multiple decoding times corresponding to the image group; each decoding time is determined by a client device decoding the image in the image group based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations.

[0103] In this embodiment, the decoding type may include hard decoding or soft decoding. In different decoding modes, the application ratio of hard decoding and soft decoding is different. For example, the image group includes 4 sub-images, a, b, c and d, then decoding mode 1 may be hard decoding of a and soft decoding of b, c and d, and decoding mode 2 may be hard decoding of a and b and soft decoding of c and d.

[0104] Specifically, continue as Figure 3 As shown, the client includes a client self-check module, which detects whether the hardware supports hard decoding based on the dedicated SDK interface of the client device through the client self-check module, and further obtains the supported hard decoding paths if it supports. According to the image combination information, each group of image combinations is subjected to parallel decoding detection of all soft decoding, all hard decoding, and mixed soft and hard decoding in different proportions, and the time consumption of each decoding mode is calculated, and the calculated time consumption is returned to the server in the form of return parameters.

[0105] For the parallel decoding mode of soft decoding and hard decoding, exemplary: for example Figure 4In the image group D shown, there are two 2048x1024 images plus one 2048x2048 image, and four decoding modes can be allocated. If hard decoding is supported, the first is that one 2048x1024 image uses a soft decoder, and one 2048x1024 and one 2048x2048 image use a hard decoder (if the number of hard decoding paths is greater than or equal to 2, the two decoding paths are processed in parallel, otherwise they are processed sequentially), and the soft and hard decoders are parallel; the second is that one 2048x2048 image uses a soft decoder, and two 2048x1024 images use a hard decoder; the third is that two 2048x1024 images use a soft decoder, and one 2048x2048 image uses a hard decoder; the fourth is that one 2048x1024 and one 2048x2048 image use a soft decoder (two images are multi-threaded in parallel), and one 2048x1024 image uses a hard decoder. Among them, for images with the same resolution, for example, two 2048x1024 images x and y, it is possible to perform soft decoding on x and hard decoding on y by default, or hard decoding on x and soft decoding on y. The time consumption of these two forms is the same, that is, only the impact of resolution on encoding and decoding time consumption is considered, and the impact of image content on encoding and decoding time consumption is ignored, thereby reducing the amount of calculation.

[0106] In some embodiments, the codec format may be fixedly adopted as the H.265 standard. If there are multiple codec formats available, the time consumption under different codec format conditions may be detected for each image combination, so as to further select the codec format according to the time consumption. For example, as shown in Table 1, the decoding time consumption T1 to T5 of different image groups (a and b) under different decoding modes (1 and 2) may be obtained.

[0107] Table 1

[0108] Codec format Image Grouping Decoding Mode Decoding time m a 1 T1 m a 2 T2 m b 1 T3 m b 2 T4 n a 1 T5

[0109] 203. For each image sub-group in the image combination information, obtain multiple encoding times corresponding to the image grouping; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations.

[0110] The encoding type may be hard coding, soft coding, or both hard coding and soft coding.

[0111] In this embodiment, continue as Figure 3As shown, the server includes a server self-check module. The server device detects whether the hardware supports hard coding and the number of hard coding paths supported through the server self-check module based on the dedicated SDK interface of the server device. According to the image combination information, parallel coding detection of all soft coding, all hard coding, and mixed soft and hard coding in different proportions is performed for each image group, and the time consumption of each coding mode is calculated. As shown in Table 2, assuming that the coding format is m, the decoding time consumption of the following different image groups under different coding modes can be obtained. It should be noted that Table 2 is only a partial display, and can also include the decoding time consumption corresponding to the remaining multiple combinations, such as the decoding time consumption T10 corresponding to the coding format m image group a coding mode 3.

[0112] Table 2

[0113] Codec format Image Grouping Encoding Mode Coding time m a 1 T6 m a 2 T7 m b 1 T8 m b 2 T9

[0114] 204. Determine a target splitting parameter, a target splicing parameter, a target decoding parameter and a target encoding parameter according to the multiple decoding time consumptions and the multiple encoding time consumptions; so that the server device splits and encodes the image to be encoded according to the target splitting parameter and the target decoding parameter to obtain the image to be decoded, and the client device decodes and splices the image to be decoded according to the target decoding parameter and the target splicing parameter to obtain the target image.

[0115] In some embodiments, determining a target splitting parameter, a target splicing parameter, a target decoding parameter, and a target encoding parameter according to the plurality of decoding times and the plurality of encoding times includes:

[0116] According to the plurality of decoding times and the plurality of encoding times, determine the sum of the decoding time and the encoding time of each image group in each encoding mode and in each decoding mode, and determine the image group corresponding to the minimum sum as the target image group, the decoding mode corresponding to the minimum sum as the target decoding mode, and the encoding mode corresponding to the minimum sum as the target encoding mode;

[0117] The target splitting parameters and the target splicing parameters are determined according to the target image grouping, the target decoding parameters are determined according to the target decoding mode, and the target encoding parameters are determined according to the target encoding mode.

[0118] Specifically, after determining multiple decoding times of different image groups in different decoding modes and multiple encoding times of different image groups in different encoding modes, the decoding time and encoding time corresponding to each combination of image grouping and encoding and decoding mode can be added as the total time corresponding to the combination, and then multiple total time costs can be obtained, so that the image group in the combination with the smallest total time cost can be determined as the target image group, the encoding mode can be determined as the target encoding mode, and the decoding mode can be determined as the target decoding mode, and then based on the target image group, the target encoding mode, and the target decoding mode, the encoding and decoding parameters, i.e., self-test parameters, are determined. The self-test parameters may include splitting parameters (such as the resolution of each image, the number of images), splicing parameters (such as the coordinate position of the upper left corner of each image on the original image), encoding parameters (such as the encoding type of each image (hard coding, soft coding, or hard coding and soft coding)) and decoding parameters (such as the decoding type of each image).

[0119] Exemplarily, by combining the data in Table 1 and Table 2, Table 3 can be obtained. As shown in Table 3, the total time consumption corresponding to each combination of image grouping, encoding mode and decoding mode can be calculated. Assuming that the sum of T4 and T9 is the smallest, image group b can be determined as the target image grouping, decoding mode 2 can be determined as the target decoding mode, and encoding mode 2 can be determined as the target encoding mode.

[0120] Table 3

[0121] Codec format Image Grouping Decoding Mode Encoding Mode Decoding time Coding time Total time m a 1 1 T1 T6 T1+T6 m a 2 2 T2 T7 T2+T7 m b 1 1 T3 T8 T3+T8 m b 2 2 T4 T9 T4+T9

[0122] In some embodiments, the method may be applied to a server device, wherein for each image group in the image combination information, obtaining multiple decoding times corresponding to the image group may include: sending the image combination information to the client device, so that the client device decodes the images in the image group based on the multiple decoding modes for each image group in the image combination information, and obtains multiple decoding times corresponding to the image group;

[0123] Receive multiple decoding times corresponding to the multiple image groups sent by the client device.

[0124] In some embodiments, the method can be applied to a server device, wherein obtaining multiple decoding times corresponding to each image group in the image combination information may include: sending the image combination information to the server device, so that the server device encodes the images in the image group based on the multiple encoding modes for each image group in the image combination information, and obtains multiple encoding times corresponding to the image group; and receiving multiple encoding times corresponding to the multiple image groups sent by the server device.

[0125] The image processing parameter determination method provided in the embodiment of the present application can fully consider the hardware conditions of the server device and the client device, and manage the encoding and decoding of the video stream by combining soft encoding and decoding with hard encoding and decoding according to the detection results of the self-detection module, so as to fully utilize the resources of the encoding device, reduce delays and freezes, maximize compatibility with the hardware conditions of the server device and the client device, and reduce the requirements and dependence on the hardware devices.

[0126] Figure 5 The following is a flow chart of the image processing method provided in the embodiment of the present application. Figure 5 As shown, the method includes:

[0127] 501. Obtain an image to be encoded and obtain splitting parameters and encoding parameters; the splitting parameters are used to split the image to be encoded into multiple sub-images; the encoding parameters include encoding types corresponding to the multiple sub-images respectively; the encoding type includes hard coding and / or soft coding.

[0128] The execution subject of this embodiment is a server device.

[0129] Specifically, the splitting parameters and the encoding parameters may be determined by using the image processing parameter determination method described in the above embodiment, such as Figure 2 The embodiment shown.

[0130] The encoding type may be hard coding, soft coding, or both hard coding and soft coding.

[0131] The splitting parameters and encoding parameters may be pre-stored in the server device after being determined, or may be pre-stored in other devices such as a database, and may be obtained from the database after the server device is started, which is not limited in this embodiment.

[0132] 502. Split the image to be encoded according to the splitting parameter to obtain multiple first images.

[0133] The splitting parameters may have various forms, and in one practicable manner may include multiple resolutions and quantities corresponding to different resolutions. Figure 4As shown, the total resolution is 4096x2048, then the splitting parameter may include two split resolutions of 2048x1024 and 2048x2048, and also include the number of corresponding to different resolutions, the number of 2048x1024 is 2, and the number of 2048x2048 is 1. In another achievable manner, the splitting parameter may include multiple coordinate values, for example, Figure 4 As shown in FIG. 1 , when the image is split into three sub-images, the coordinates of the four corners of the three sub-images can be recorded. In one possible implementation, the splitting parameters can include a bounding box, such as Figure 4 As shown, the bounding box formed by the dotted lines can be enlarged or reduced as the total resolution of the image changes, so that no matter what the total resolution of the image to be segmented is, the image can be segmented based on the bounding box.

[0134] In some embodiments, the splitting parameters include at least one group of resolutions and a preset number corresponding to each group of resolutions; each group of resolutions has the same resolution value, and different groups of resolutions have different resolution values; splitting the image to be encoded according to the splitting parameters to obtain multiple first images may include: splitting the image to be encoded according to the at least one group of resolutions and the preset numbers corresponding to the at least one group of resolutions to obtain multiple first images; the multiple first images include a corresponding preset number of images under each of the resolutions.

[0135] 503. Encode the multiple first images according to the multiple encoding types to obtain multiple images to be decoded, and send the multiple images to be decoded to a client device; the multiple first images and the multiple images to be decoded are in a one-to-one correspondence.

[0136] Specifically, after the self-check is completed, that is, after the image processing parameters are determined, that is, the self-check parameters are obtained (which may include splitting parameters (such as the resolution of each image, the number of images), splicing parameters (such as the coordinate position of the upper left corner of each image on the original image), encoding parameters (such as the encoding type of each image) and decoding parameters (such as the decoding type of each image)), the server device starts to render the cloud VR and obtain the first frame image (such as a binocular image), and then passes the image to its own image processing module. The image processing module performs image splitting on the first frame image based on the self-check parameters (that is, the determined encoding parameters, splitting parameters, etc.) to obtain a new image corresponding to the resolution and number in the self-check parameters. Then, according to the allocation method of hard coding and soft coding indicated by the encoding mode in the self-check parameters (such as which images are hard-coded and which images are soft-coded), the new images are submitted to the soft and hard coding modules for parallel encoding (here, parallel includes soft coding and hard coding at the same time). Finally, the encoded new image data is packaged and sent to the terminal. Optionally, the data packaging content may include self-check parameters (the encoding parameters may not be included because the encoding parameters will not be used in the decoding process) and each encoded image.

[0137] In some embodiments, the encoding type includes hard coding and / or soft coding, and encoding the multiple first images according to the multiple encoding types to obtain multiple images to be decoded may include: if there are images to be hard-coded whose corresponding encoding type is hard coding among the multiple first images, and there are images to be soft-coded whose corresponding encoding type is soft coding, then hard coding of the images to be hard-coded and soft coding of the images to be soft-coded are performed in parallel to obtain multiple images to be decoded.

[0138] Specifically, the multiple first images respectively correspond to encoding types allocated based on encoding parameters. Exemplarily, the multiple first images can all be hard-coded, or all be soft-coded, or some first images can be hard-coded and some can be soft-coded.

[0139] In some embodiments, in order to improve the coding efficiency, multiple hard coding paths can be performed in parallel, and the coding parameters also include the number of hard coding paths supported by the server device; the hard coding of the image to be hard-coded and the soft coding of the image to be soft-coded are performed in parallel to obtain multiple images to be decoded, including: if the number of hard coding paths is multiple, then when hard coding the image to be hard-coded, the image to be hard-coded is divided into multiple paths for parallel hard coding according to the number of hard coding paths.

[0140] In some embodiments, encoding the multiple first images according to the multiple encoding types may include: acquiring an encoding format; and encoding the multiple first images according to the multiple encoding types and the encoding format.

[0141] The image processing method provided in this embodiment splits the images in the video stream, and hard codes some of the split images, while soft codes the other parts of the images. The encoding of the video stream can be completed by combining soft coding and hard coding, so as to make full use of the resources of the encoding device, reduce delays and freezes, and improve the user experience.

[0142] Figure 6 The following is a flow chart of the image processing method provided in the embodiment of the present application. Figure 6 As shown, the method includes:

[0143] 601. Acquire multiple images to be decoded sent by a server device; the multiple images to be decoded are obtained by the server device based on splitting parameters and encoding parameters; the splitting parameters are used to split the images to be encoded into multiple sub-images; the encoding parameters include encoding types corresponding to the multiple sub-images respectively.

[0144] The execution subject of this embodiment may be a client device.

[0145] Specifically, after the server device completes the splitting and encoding of the image to be encoded based on the splitting parameters and encoding parameters in the self-test parameters (based on the encoding and decoding parameters determined in the above embodiment), it obtains multiple images to be decoded and sends the multiple images to be decoded to the client device.

[0146] 602. Obtain decoding parameters and splicing parameters; the decoding parameters include decoding types corresponding to the multiple sub-images respectively; the splicing parameters are used to characterize the relative position relationship of the multiple sub-images.

[0147] Specifically, the decoding parameters and splicing parameters may be packaged together with multiple images to be decoded and then sent from the server device to the client device, or may be sent separately by the server device, or may be pre-stored in the client device.

[0148] 603. Decode the multiple images to be decoded according to the multiple decoding types to obtain multiple second images.

[0149] 604. Stitch the multiple second images according to the stitching parameters to obtain a target image.

[0150] Exemplarily, the stitching parameters may include position information of different sub-images relative to the original image, such as the coordinates of the upper left corner of the sub-image in the original image.

[0151] Specifically, the client decodes each image in parallel with its corresponding decoding type (soft decoding or hard decoding) based on the decoding parameters (resolution and decoding type description) in the self-check parameters of the server, and transmits the decoded image and self-check parameters (including stitching parameters) to the image reconstruction module of the client. The image reconstruction module re-renders each decoded image to the final target image based on the stitching parameters in the self-check parameters, such as the coordinate position of the upper left corner of each image on the original image, and the client uses the target image to present it on the display screen.

[0152] In some embodiments, the decoding type includes hard decoding or soft decoding; decoding the multiple images to be decoded according to the multiple decoding types to obtain multiple second images may include: if there are images to be hard-decoded whose corresponding decoding type is hard decoding among the multiple images to be decoded, and there are images to be soft-decoded whose corresponding decoding type is soft decoding, then hard decoding of the images to be hard-decoded and soft decoding of the images to be soft-decoded are performed in parallel to obtain multiple second images.

[0153] In some embodiments, in order to improve efficiency, multiple hard decodings can be performed in parallel. The decoding parameters also include the number of hard decoding paths supported by the client device; the hard decoding of the image to be hard-decoded and the soft decoding of the image to be soft-decoded are performed in parallel to obtain multiple second images, which may include: if the number of hard decoding paths is multiple, then when hard decoding the image to be hard-decoded, the image to be hard-decoded is divided into multiple paths for parallel hard decoding according to the number of hard decoding paths.

[0154] In some embodiments, the stitching parameters include position information of each sub-image on the image to be encoded; stitching the multiple second images according to the stitching parameters to obtain a target image may include: stitching the multiple second images according to position information of the sub-images respectively corresponding to the multiple second images on the image to be encoded to obtain the target image.

[0155] The image processing method provided in this embodiment splits the images in the video stream, and hard codes some of the split images and soft codes the other images, and then hard decodes some of the encoded images on the client and soft decodes the other images. Thus, the encoding and decoding of the video stream can be completed by combining soft encoding and decoding with hard encoding and decoding, so as to make full use of the encoding and decoding equipment resources, reduce delays and freezes, and improve the user experience.

[0156] Figure 7 This is a schematic diagram of the structure of the server device provided in the embodiment of the present application. Figure 7As shown, the server device 70 includes: a first acquisition module 701, a splitting module 702 and an encoding module 703.

[0157] The first acquisition module 701 is used to acquire an image to be encoded and to acquire a splitting parameter and an encoding parameter; the splitting parameter is used to split the image to be encoded into multiple sub-images; the encoding parameter includes encoding types corresponding to the multiple sub-images respectively; the encoding type includes hard coding and / or soft coding;

[0158] A splitting module 702, configured to split the image to be encoded according to the splitting parameter to obtain a plurality of first images;

[0159] The encoding module 703 is used to encode the multiple first images according to the multiple encoding types to obtain multiple images to be decoded, so as to send the multiple images to be decoded to the client device; the multiple first images and the multiple images to be decoded are in a one-to-one correspondence.

[0160] The server device provided in the embodiment of the present application splits the images in the video stream, and uses hard coding on some of the split images and soft coding on the other part of the images. It can complete the encoding of the video stream by combining soft coding and hard coding, so as to fully utilize the resources of the encoding device, reduce delays and freezes, and improve the user experience.

[0161] In some embodiments, the splitting module 702 is specifically used to: split the image to be encoded according to the at least one group of resolutions and the preset numbers corresponding to the at least one group of resolutions, to obtain multiple first images; the multiple first images include the corresponding preset number of images under each of the resolutions.

[0162] In some embodiments, the encoding type includes hard coding and / or soft coding, and the encoding module 703 is specifically used to: if there are images to be hard-coded whose corresponding encoding type is hard coding among the multiple first images, and there are images to be soft-coded whose corresponding encoding type is soft coding, then the hard coding of the images to be hard-coded and the soft coding of the images to be soft-coded are performed in parallel to obtain multiple images to be decoded.

[0163] In some embodiments, the encoding parameters also include the hard-coding paths supported by the server device; the encoding module 703 is specifically used to: if the hard-coding path is multiple, then when hard-coding the image to be hard-coded, the image to be hard-coded is divided into multiple paths for parallel hard coding according to the hard-coding path.

[0164] In some embodiments, the encoding module 703 is specifically used to: obtain an encoding format; and encode the multiple first images according to the multiple encoding types and the encoding format.

[0165] The server device provided in the embodiment of the present application can be used to execute the above-mentioned image processing method embodiment with the server device as the execution subject. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0166] Figure 8 This is a schematic diagram of the structure of the client device provided in the embodiment of the present application. Figure 8 As shown, the client device 80 includes: a receiving module 801 , a second obtaining module 802 , a decoding module 803 and a splicing module 804 .

[0167] The receiving module 801 is used to obtain multiple images to be decoded sent by the server device; the multiple images to be decoded are obtained by the server device based on the splitting parameter and the encoding parameter; the splitting parameter is used to split the image to be encoded into multiple sub-images; the encoding parameter includes the encoding types corresponding to the multiple sub-images respectively;

[0168] The second acquisition module 802 is used to acquire decoding parameters and splicing parameters; the decoding parameters include decoding types corresponding to the multiple sub-images respectively; the splicing parameters are used to characterize the relative position relationship of the multiple sub-images;

[0169] A decoding module 803, configured to decode the multiple images to be decoded according to the multiple decoding types to obtain multiple second images;

[0170] The stitching module 804 is used to stitch the multiple second images according to the stitching parameters to obtain a target image.

[0171] The server device provided in the embodiment of the present application splits the images in the video stream, and hard codes some of the split images and soft codes the other images, and then hard decodes some of the encoded images on the client side and soft decodes the other images. Thus, the encoding and decoding of the video stream can be completed by combining soft encoding and decoding with hard encoding and decoding, so as to make full use of the encoding and decoding equipment resources, reduce delays and freezes, and improve the user experience.

[0172] In some embodiments, the decoding type includes hard decoding or soft decoding; the decoding module 803 is specifically used for: if there is an image to be hard-decoded whose corresponding decoding type is hard decoding among the multiple images to be decoded, and there is an image to be soft-decoded whose corresponding decoding type is soft decoding, then the hard decoding of the image to be hard-decoded and the soft decoding of the image to be soft-decoded are performed in parallel to obtain multiple second images.

[0173] In some embodiments, the decoding parameters also include the number of hard decoding paths supported by the client device; the decoding module 803 is specifically used for: if the number of hard decoding paths is multiple, then when hard decoding the image to be hard-decoded, the image to be hard-decoded is divided into multiple paths for parallel hard decoding according to the number of hard decoding paths.

[0174] In some embodiments, the stitching parameters include position information of each sub-image on the image to be encoded; the stitching module 804 is specifically used to: stitch the multiple second images according to the position information of the sub-images respectively corresponding to the multiple second images on the image to be encoded to obtain the target image.

[0175] The client device provided in the embodiment of the present application can be used to execute the above-mentioned image processing method embodiment with the client device as the execution subject. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0176] Fig. 9 This is a schematic diagram of the structure of the image processing parameter determination device provided in the embodiment of the present application. Fig. 9 As shown, the image codec determination device 90 includes: a third acquisition module 901 , a fourth acquisition module 902 , a fifth acquisition module 903 and a determination module 904 .

[0177] The third acquisition module 901 is used to acquire image combination information; the image combination information includes multiple image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters.

[0178] The fourth acquisition module 902 is used to obtain multiple decoding times corresponding to each image group in the image combination information; each decoding time is determined by the client device decoding the image in the image group based on the corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations.

[0179] The fifth acquisition module 903 is used to obtain multiple encoding times corresponding to each image grouping in the image combination information; each encoding time is determined by encoding the image in the image group by the server device based on the corresponding encoding mode; each encoding mode includes the encoding type used for each sub-image of the image group, and different encoding modes have different encoding type combinations.

[0180] The determination module 904 is used to determine the target splitting parameters, the target splicing parameters, the target decoding parameters and the target encoding parameters according to the multiple decoding time consumptions and the multiple encoding time consumptions; so that the server device splits and encodes the image to be encoded according to the target splitting parameters and the target decoding parameters to obtain the image to be decoded, and the client device decodes and splices the image to be decoded according to the target decoding parameters and the target splicing parameters to obtain the target image.

[0181] The server device provided in the embodiment of the present application can fully consider the hardware conditions of the server device and the client device, and manage the encoding and decoding of the video stream by combining soft encoding and decoding with hard encoding and decoding according to the detection results of the self-detection module, so as to fully utilize the resources of the encoding device, reduce delays and freezes, maximize compatibility with the hardware conditions of the server device and the client device, and reduce the requirements and dependence on hardware devices.

[0182] In some embodiments, the determination module 904 is specifically used to: determine the sum of the decoding time and the encoding time of each image group in each encoding mode and in each decoding mode based on multiple decoding times and multiple encoding times, and determine the image group corresponding to the minimum sum as the target image group, determine the decoding mode corresponding to the minimum sum as the target decoding mode, and determine the encoding mode corresponding to the minimum sum as the target encoding mode; determine the target splitting parameters and the target splicing parameters according to the target image group, determine the target decoding parameters according to the target decoding mode, and determine the target encoding parameters according to the target encoding mode.

[0183] In some embodiments, applied to a server device, the fourth acquisition module 902 is specifically used to: send the image combination information to the client device, so that the client device decodes the images in the image group based on the multiple decoding modes for each image group in the image combination information, and obtains multiple decoding times corresponding to the image group; and receives multiple decoding times corresponding to the multiple image groups sent by the client device.

[0184] In some embodiments, applied to a server device, the fifth acquisition module 903 is specifically used to: send the image combination information to the server device, so that the server device encodes the images in the image group based on the multiple encoding modes for each image group in the image combination information, and obtains multiple encoding times corresponding to the image group; and receives multiple encoding times corresponding to the multiple image groups sent by the server device.

[0185] The image processing parameter determination device provided in the embodiment of the present application can be used to execute the above-mentioned encoding and decoding parameter determination method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0186] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The device can be used as a server device or a client device. The device can be a computer, VR device, tablet computer, mobile phone, server, etc.

[0187] The device 120 may include one or more of the following components: a processing component 1201 , a memory 1202 , a power component 1203 , a multimedia component 1204 , an audio component 1205 , an input / output (I / O) interface 1206 , a sensor component 1207 , and a communication component 1208 .

[0188] The processing component 1201 generally controls the overall operation of the device 120, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1201 may include one or more processors 1209 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1201 may include one or more modules to facilitate the interaction between the processing component 1201 and other components. For example, the processing component 1201 may include a multimedia module to facilitate the interaction between the multimedia component 1204 and the processing component 1201.

[0189] The memory 1202 is configured to store various types of data to support operations on the device 120. Examples of such data include instructions for any application or method operating on the device 120, contact data, phone book data, messages, pictures, videos, etc. The memory 1202 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0190] The power supply component 1203 provides power to the various components of the device 120. The power supply component 1203 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 120.

[0191] The multimedia component 1204 includes a screen that provides an output interface between the device 120 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1204 includes a front camera and / or a rear camera. When the device 120 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0192] The audio component 1205 is configured to output and / or input audio signals. For example, the audio component 1205 includes a microphone (MIC), and when the device 120 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1202 or sent via the communication component 1208. In some embodiments, the audio component 1205 also includes a speaker for outputting audio signals.

[0193] I / O interface 1206 provides an interface between processing component 1201 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0194] The sensor assembly 1207 includes one or more sensors for providing various aspects of status assessment for the device 120. For example, the sensor assembly 1207 can detect the open / closed state of the device 120, the relative positioning of components, such as the display and keypad of the device 120, the sensor assembly 1207 can also detect the position change of the device 120 or a component of the device 120, the presence or absence of user contact with the device 120, the orientation or acceleration / deceleration of the device 120, and the temperature change of the device 120. The sensor assembly 1207 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1207 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1207 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0195] The communication component 1208 is configured to facilitate wired or wireless communication between the device 120 and other devices. The device 120 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1208 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1208 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0196] In an exemplary embodiment, the device 120 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0197] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1202 including instructions, and the instructions can be executed by a processor 1209 of the device 120 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0198] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0199] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0200] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0201] An embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method performed by the above image encoding device.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: Applied to a server device, the method comprises: Acquire an image to be encoded and acquire a splitting parameter and an encoding parameter; the splitting parameter is used to split the image to be encoded into multiple sub-images; the encoding parameter includes encoding types corresponding to the multiple sub-images respectively; the encoding type includes hard coding and / or soft coding; Splitting the to-be-encoded image according to the splitting parameter to obtain a plurality of first images; According to the multiple encoding types, the multiple first images are encoded to obtain multiple images to be decoded, so as to send the multiple images to be decoded to a client device; the multiple first images and the multiple images to be decoded are in a one-to-one correspondence.

2. The method according to claim 1, characterized in that The splitting parameters include at least one set of resolutions and a preset number corresponding to each set of resolutions; each set of resolutions has the same resolution value, and different sets of resolutions have different resolution values; The step of splitting the image to be encoded according to the splitting parameter to obtain a plurality of first images comprises: Splitting the image to be encoded according to the at least one set of resolutions and the preset numbers respectively corresponding to the at least one set of resolutions to obtain a plurality of first images; The plurality of first images include a corresponding preset number of images at each of the resolutions.

3. The method according to claim 1, characterized in that The step of encoding the plurality of first images according to the plurality of encoding types to obtain a plurality of images to be decoded comprises: If there are images to be hard-coded whose corresponding encoding type is hard coding among the multiple first images, and there are images to be soft-coded whose corresponding encoding type is soft coding, hard coding of the images to be hard-coded and soft coding of the images to be soft-coded are performed in parallel to obtain multiple images to be decoded.

4. The method according to claim 3, characterized in that The encoding parameters also include the number of hard encoding paths supported by the server device; the hard encoding of the image to be hard-coded and the soft encoding of the image to be soft-coded are performed in parallel to obtain a plurality of images to be decoded, including: If the number of hard-coding paths is multiple, when hard-coding the image to be hard-coded, the image to be hard-coded is divided into multiple paths for parallel hard coding according to the number of hard-coding paths.

5. The method according to any one of claims 1 to 4, characterized in that: The step of encoding the plurality of first images according to the plurality of encoding types comprises: Get the encoding format; The plurality of first images are encoded according to the plurality of encoding types and the encoding format.

6. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the image to be encoded and obtaining the splitting parameters and encoding parameters, the method further includes: Acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters; For each image group in the image combination information, a plurality of decoding times corresponding to the image group is obtained; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations; For each image subassembly group in the image combination information, a plurality of encoding times corresponding to the image group is obtained; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations; The splitting parameter and the encoding parameter are determined according to the multiple decoding times and the multiple encoding times.

7. An image processing method, characterized in that: Applied to a client device, the method comprises: Acquire multiple images to be decoded sent by the server device; the multiple images to be decoded are obtained by the server device based on splitting parameters and encoding parameters; the splitting parameters are used to split the image to be encoded into multiple sub-images; the encoding parameters include encoding types corresponding to the multiple sub-images respectively; Obtaining decoding parameters and splicing parameters; the decoding parameters include decoding types corresponding to the multiple sub-images respectively; the splicing parameters are used to characterize the relative position relationship of the multiple sub-images; Decoding the multiple images to be decoded according to the multiple decoding types to obtain multiple second images; The plurality of second images are stitched together according to the stitching parameters to obtain a target image.

8. The method according to claim 7, characterized in that The decoding type includes hard decoding or soft decoding; and decoding the multiple images to be decoded according to the multiple decoding types to obtain multiple second images includes: If there are images to be hard-decoded whose corresponding decoding type is hard decoding among the multiple images to be decoded, and there are images to be soft-decoded whose corresponding decoding type is soft decoding, hard decoding of the images to be hard-decoded and soft decoding of the images to be soft-decoded are performed in parallel to obtain multiple second images.

9. The method according to claim 8, characterized in that The decoding parameters also include the number of hard decoding paths supported by the client device; the hard decoding of the image to be hard-decoded and the soft decoding of the image to be soft-decoded are performed in parallel to obtain a plurality of second images, including: If the number of hard decoding paths is multiple, when hard decoding is performed on the image to be hard-decoded, the image to be hard-decoded is divided into multiple paths for parallel hard decoding according to the number of hard decoding paths.

10. The method according to any one of claims 7 to 9, characterized in that: The splicing parameters include position information of each sub-image on the image to be encoded; splicing the plurality of second images according to the splicing parameters to obtain a target image, including: The multiple second images are spliced ​​according to the position information of the sub-images respectively corresponding to the multiple second images on the image to be encoded to obtain a target image.

11. The method according to any one of claims 7 to 9, characterized in that: Before obtaining the decoding parameters and the splicing parameters, the method further includes: Acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters; For each image group in the image combination information, a plurality of decoding times corresponding to the image group is obtained; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations; For each image subassembly group in the image combination information, a plurality of encoding times corresponding to the image group is obtained; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations; The splicing parameter and the decoding parameter are determined according to the multiple decoding time consumptions and the multiple encoding time consumptions.

12. A method for determining image encoding and decoding parameters, characterized in that: include: Acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters; For each image group in the image combination information, a plurality of decoding times corresponding to the image group is obtained; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes a decoding type respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations; For each image subassembly group in the image combination information, a plurality of encoding times corresponding to the image group is obtained; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations; According to the multiple decoding times and the multiple encoding times, target splitting parameters, target splicing parameters, target decoding parameters and target encoding parameters are determined; so that the server device splits and encodes the image to be encoded according to the target splitting parameters and the target decoding parameters to obtain the image to be decoded, and the client device decodes and splices the image to be decoded according to the target decoding parameters and the target splicing parameters to obtain the target image.

13. The method according to claim 12, characterized in that The determining of a target splitting parameter, a target splicing parameter, a target decoding parameter, and a target encoding parameter according to the plurality of decoding times and the plurality of encoding times comprises: According to the plurality of decoding times and the plurality of encoding times, determine the sum of the decoding time and the encoding time of each image group in each encoding mode and in each decoding mode, and determine the image group corresponding to the minimum sum as the target image group, the decoding mode corresponding to the minimum sum as the target decoding mode, and the encoding mode corresponding to the minimum sum as the target encoding mode; The target splitting parameters and the target splicing parameters are determined according to the target image grouping, the target decoding parameters are determined according to the target decoding mode, and the target encoding parameters are determined according to the target encoding mode.

14. The method according to claim 12 or 13, characterized in that Applied to a server device, the obtaining, for each image group in the image combination information, a plurality of decoding times corresponding to the image group includes: Sending the image combination information to the client device, so that the client device decodes the images in the image group based on the multiple decoding modes for each image group in the image combination information, and obtains multiple decoding times corresponding to the image group; Receive multiple decoding times corresponding to the multiple image groups sent by the client device.

15. The method according to claim 12 or 13, characterized in that Applied to a server device, the obtaining, for each image group in the image combination information, a plurality of decoding times corresponding to the image group includes: Sending the image combination information to the server device, so that the server device encodes the images in the image group based on the multiple encoding modes for each image group in the image combination information, and obtains multiple encoding times corresponding to the image group; Receiving multiple encoding times corresponding to the multiple image groups sent by the server device.

16. A server device, characterized in that: include: A first acquisition module, used to acquire the image to be encoded and to acquire splitting parameters and encoding parameters; The splitting parameter is used to split the to-be-encoded image into a plurality of sub-images; The encoding parameters include encoding types corresponding to the multiple sub-images respectively; the encoding types include hard coding and / or soft coding; A splitting module, used for splitting the image to be encoded according to the splitting parameter to obtain a plurality of first images; The encoding module is used to encode the multiple first images according to the multiple encoding types to obtain multiple images to be decoded, so as to send the multiple images to be decoded to the client device; the multiple first images and the multiple images to be decoded are in a one-to-one correspondence.

17. A client device, characterized in that: include: A receiving module, used for acquiring a plurality of images to be decoded sent by a server device; the plurality of images to be decoded are obtained by the server device based on splitting parameters and encoding parameters; The splitting parameter is used to split the image to be encoded into multiple sub-images; The encoding parameters include encoding types corresponding to the multiple sub-images respectively; A second acquisition module is used to acquire decoding parameters and splicing parameters; the decoding parameters include decoding types corresponding to the multiple sub-images respectively; The stitching parameters are used to characterize the relative position relationship of the multiple sub-images; A decoding module, configured to decode the plurality of images to be decoded according to the plurality of decoding types to obtain a plurality of second images; The stitching module is used to stitch the multiple second images according to the stitching parameters to obtain a target image.

18. An image processing parameter determination device, characterized in that: include: A third acquisition module is used to acquire image combination information; the image combination information includes a plurality of image groups; sub-images in different image groups are obtained by splitting the original image according to different splitting parameters; a fourth acquisition module, configured to acquire, for each image group in the image combination information, a plurality of decoding times corresponding to the image group; each decoding time is determined by decoding the image in the image group by a client device based on a corresponding decoding mode; each decoding mode includes decoding types respectively used for each sub-image of the image group, and different decoding modes have different decoding type combinations; a fifth acquisition module, configured to acquire, for each image sub-group in the image combination information, a plurality of encoding times corresponding to the image group; each encoding time is determined by encoding the image in the image group by a server device based on a corresponding encoding mode; each encoding mode includes encoding types respectively used for each sub-image of the image group, and different encoding modes have different encoding type combinations; A determination module, used to determine a target splitting parameter, a target splicing parameter, a target decoding parameter and a target encoding parameter according to the plurality of decoding times and the plurality of encoding times; The server device splits and encodes the image to be encoded according to the target splitting parameters and the target decoding parameters to obtain the image to be decoded, and the client device decodes and splices the image to be decoded according to the target decoding parameters and the target splicing parameters to obtain the target image.

19. A server device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 15.

20. A client device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1-13 and 15.

21. An image processing system, characterized in that It comprises the server device as claimed in claim 19 and the client device as claimed in claim 20.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 15 is implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Citation Information

Patent Citations

  • WebRTC data coding method, device, and computer readable storage medium

    CN108156414A

  • Cloud application picture display method and device, server, terminal and storage medium

    CN115202550A

  • Image and video coding and decoding

    GB202308546D0

  • Distributed and parallel video stream encoding and transcoding

    US20200029086A1

  • Video posting method and apparatus, and device and storage medium

    WO2023174254A1