Encoding methods, electronic devices, communication systems, storage media, and program products

Abstract

This application provides an encoding method, an electronic device, a communication system, a storage medium, and a program product. The encoding method includes: a first device determining the I-frame positions of a first image stream and a second image stream based on a first resolution, a second resolution, an I-frame time interval, and a frame rate; the first device encoding the first image stream according to its I-frame position to obtain a first compressed stream; the first device encoding the second image stream according to its I-frame position to obtain a second compressed stream; and the first device sending the first compressed stream and the second compressed stream to a second device. This method can solve the latency problem caused by long transmission times of multiple image streams.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an encoding method, electronic device, communication system, storage medium and program product. Background Technology

[0002] When a local device needs to obtain multiple image streams from a remote device, it can control one remote device to activate multiple cameras via the network. The remote device then sends the image streams captured by each camera to the local device, allowing the user to view the image streams captured by multiple cameras on the remote device on the local device. Alternatively, the local device can control multiple remote devices to activate their cameras via the network, and the multiple remote devices will send their captured image streams to the local device, allowing the user to view the image streams captured by multiple remote devices on the local device.

[0003] However, each image stream has a large amount of data, and the time it takes to transmit multiple image streams to the local device is long, resulting in a large delay in the display of the image on the local device. Summary of the Invention

[0004] This application provides an encoding method, electronic device, communication system, storage medium, and program product to solve the latency problem caused by long transmission times of multiple image streams.

[0005] In a first aspect, an encoding method provided in this application is applied to a communication system, the communication system comprising: a first device and a second device, the first device comprising: a first camera and a second camera, the method comprising:

[0006] The second device sends a first instruction to the first device;

[0007] The first device configures a first resolution and a second resolution according to the first instruction; the first device controls the first camera and the second camera to acquire images; the first device processes the image stream acquired by the first camera to obtain a first image stream, the resolution of the first image stream being the first resolution; the first device processes the image stream acquired by the second camera to obtain a second image stream, the resolution of the second image stream being the second resolution; the first device determines the I-frame position of the first image stream and the I-frame position of the second image stream according to the first resolution, the second resolution, the I-frame time interval, and the frame rate; the first device encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream; the first device encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream; the first device sends the first compressed stream and the second compressed stream to the second device.

[0008] Because the first device compresses the two image streams separately, the data volume of the two compressed streams is much smaller than that of the corresponding image streams. The transmission time from the two compressed streams to the second device is shorter. Upon receiving the two compressed streams, the second device can decode them to obtain the corresponding image streams, thus solving the latency problem caused by the long transmission time of multiple image streams. Furthermore, before compressing the two image streams separately, the first device calculates the I-frame positions of the two image streams based on the resolution. For each image stream, an I-frame is generated at the corresponding position, ensuring an even distribution of I-frames between the two image streams. This avoids the image jitter problem caused by the instantaneous increase in data volume when I-frames are concurrently transmitted.

[0009] The encoding method provided in the first aspect, the first device is also referred to as a remote device in this application embodiment, and the second device is also referred to as a local device in this application embodiment.

[0010] In one possible implementation, the I-frame position is the frame number of the I-frame; the first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, including: the first device uses the frame number of the first frame image in the first image stream as the frame number of the first I-frame of the first image stream, and uses the frame number of the first frame image in the second image stream as the frame number of the first I-frame of the second image stream; the first device determines the I-frame position weight of the first image stream and the I-frame position weight of the second image stream based on the first resolution and the second resolution; the first device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate, where n is greater than or equal to 2.

[0011] In one possible implementation, the first device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate, including: the first device determining the frame number of the second I-frame of the first image stream; the first device determining the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate; the first device determining the frame number of the mth I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3; and the first device determining the frame number of the mth I-frame of the second image stream based on the frame number of the second I-frame of the second image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3.

[0012] In one possible implementation, the first device determines the I-frame position weights of the first image stream and the second image stream based on the first resolution and the second resolution, including: the first device determines the I-frame position weights of the first image stream using the following formula:

[0013]

[0014] Wherein, W1 is the I-frame position weight of the first image stream, S1 is the first resolution, and S2 is the second resolution;

[0015] The first device determines the I-frame position weights of the second image stream using the following formula:

[0016]

[0017] Wherein, W2 is the I-frame position weight of the second image stream, S1 is the first resolution, and S2 is the second resolution.

[0018] In one possible implementation, the first device determines the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: the first device determines the frame number of the second I-frame of the second image stream using the following formula:

[0019] P2 = P1 + (W1 + W2) * INR * Fps / 2

[0020] Wherein, P2 is the frame number of the second I-frame of the second image stream, P1 is the frame number of the second I-frame of the first image stream, W1 is the I-frame position weight of the first image stream, W2 is the I-frame position weight of the second image stream, INR is the I-frame time interval, and Fps is the frame rate.

[0021] In one possible implementation, the first device determines the frame number of the m-th I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: the first device determines the frame number of the m-th I-frame of the first image stream using the following formula:

[0022] P m =P m-1 +INR*Fps

[0023] Among them, P m P is the frame number of the m-th I-frame in the first image stream. m-1 INR is the frame number of the (m-1)th I-frame of the first image stream, Fps is the frame rate;

[0024] The first device determines the frame number of the m-th I-frame of the second image stream based on the frame number of the second I-frame, the I-frame time interval, and the frame rate, including: the first device determines the frame number of the m-th I-frame of the second image stream using the following formula:

[0025] P′ m =P′ m-1 +INR*Fps

[0026] Among them, P′ m Let P' be the frame number of the m-th I-frame in the second image stream. m-1 INR is the frame number of the (m-1)th I-frame of the second image stream, Fps is the frame rate.

[0027] In one possible implementation, before the first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, the method further includes: the first device determining the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, wherein the first bandwidth threshold is greater than the second bandwidth threshold.

[0028] In one possible implementation, the first device determines the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, including: if the current network bandwidth is greater than or equal to the first bandwidth threshold, the first device uses a first preset interval as the I-frame time interval; if the current network bandwidth is greater than the second bandwidth threshold and less than the first bandwidth threshold, the first device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, wherein the second preset interval is greater than the first preset interval.

[0029] In one possible implementation, the first device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, including: the first device determines the I-frame time interval using the following formula:

[0030] INR = INR normal +(INR max -INR normal )*(BW-TH2) / (TH1-TH2)

[0031] Wherein, INR is the I-frame time interval, INR normal For the first preset interval, INR max The second preset interval is defined as BW, the current network bandwidth is defined as TH1, the first bandwidth threshold is defined as TH2, and the second bandwidth threshold is defined as TH2.

[0032] In one possible implementation, the I-frame position is the timestamp of the I-frame; the first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, including: the first device determines the frame sequence number of the I-frame of the first image stream and the frame sequence number of the I-frame of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate; the first device determines the timestamp of the I-frame of the first image stream based on the frame sequence number of the I-frame of the first image stream, the timestamp of the first frame image in the first image stream, and the frame rate; the first device determines the timestamp of the I-frame of the second image stream based on the frame sequence number of the I-frame of the second image stream, the timestamp of the first frame image in the second image stream, and the frame rate.

[0033] In one possible implementation, the first device further includes a third camera, and the method further includes: the second device sending a second instruction to the first device; the first device configuring a third resolution according to the second instruction; the first device controlling the third camera to acquire images; the first device processing the image stream acquired by the third camera to obtain a third image stream, the resolution of the third image stream being the third resolution; the first device determining the I-frame position of the first image stream, the I-frame position of the second image stream, and the I-frame position of the third image stream based on the first resolution, the second resolution, the third resolution, the I-frame time interval, and the frame rate.

[0034] In one possible implementation, the first device determines the I-frame positions of the first image stream, the second image stream, and the third image stream based on the first resolution, the second resolution, the third resolution, the I-frame time interval, and the frame rate. This includes: when the first device captures the first frame image using the third camera, updating the frame numbers of the images captured by the first camera and the second camera to the frame number of the first frame image; the first device uses the frame number of the first frame image in the first image stream as the frame number of the first I-frame of the first image stream, and uses the frame number of the first frame image in the second image stream as the frame number of the first I-frame of the second image stream, and sets the frame number of the first frame image in the second image stream as the frame number of the first I-frame of the second image stream. The frame number of the first frame in the third image stream is used as the frame number of the first I-frame in the third image stream; the first device determines the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, and the I-frame position weight of the third image stream based on the first resolution, the second resolution, and the third resolution; the first device determines the frame number of the nth I-frame of the first image stream, the frame number of the nth I-frame of the second image stream, and the frame number of the nth I-frame of the third image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame position weight of the third image stream, the I-frame time interval, and the frame rate, where n is greater than or equal to 2.

[0035] In one possible implementation, the method further includes: the second device sending a third instruction to the first device; the first device configuring a fourth resolution according to the third instruction; the first device processing the image stream captured by the first camera to obtain a fourth image stream, the resolution of the fourth image stream being the fourth resolution; and the first device determining the I-frame position of the second image stream and the I-frame position of the fourth image stream according to the second resolution, the fourth resolution, the I-frame time interval, and the frame rate.

[0036] In one possible implementation, the first device determines the I-frame position of the second image stream and the I-frame position of the fourth image stream based on the second resolution, the fourth resolution, the I-frame time interval, and the frame rate, including: the first device updating the frame number of the images captured by the first camera and the second camera when the third instruction is received to the frame number of the first frame image; the first device using the frame number of the first frame image in the second image stream as the frame number of the first I-frame of the first image stream, and using the frame number of the first frame image in the fourth image stream as the frame number of the first I-frame of the fourth image stream; the first device determining the I-frame position weight of the second image stream and the I-frame position weight of the fourth image stream based on the second resolution and the fourth resolution; and the first device determining the frame number of the nth I-frame of the second image stream and the frame number of the nth I-frame of the fourth image stream based on the I-frame position weight of the second image stream, the I-frame position weight of the fourth image stream, the I-frame time interval, and the frame rate, where n is greater than or equal to 2.

[0037] Secondly, embodiments of this application provide an encoding method applied to a communication system, the communication system comprising: a first device, a second device, and a third device, the method comprising:

[0038] The third device sends a first instruction to the first device and the second device respectively;

[0039] The first device configures a first resolution according to the first instruction; processes the image stream acquired by the first device to obtain a first image stream, wherein the resolution of the first image stream is the first resolution;

[0040] The second device configures a second resolution according to the first instruction; processes the image stream acquired by the second device to obtain a second image stream, the resolution of which is the second resolution;

[0041] The third device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate; sends the I-frame position of the first image stream to the first device; and sends the I-frame position of the second image stream to the second device.

[0042] The first device encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream; and sends the first compressed stream to the third device.

[0043] The second device encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream; and sends the second compressed stream to the third device.

[0044] Because the first device compresses the first image stream and the second device compresses the second image stream, the data volume of the two compressed streams is much smaller than the data volume of the corresponding image streams. The transmission time of the two compressed streams to the third device is shorter. After receiving the two compressed streams, the third device decodes them to obtain the corresponding image streams, thus solving the latency problem caused by the long transmission time of multiple image streams. Furthermore, the third device calculates the I-frame positions of the two image streams based on the resolution and sends these positions to the corresponding devices. This allows the first device to generate an I-frame at the corresponding position based on the received I-frame position, and the second device to generate an I-frame at the corresponding position based on the received I-frame position. Since the I-frames of the two image streams are evenly distributed, the image jitter problem caused by the instantaneous increase in data volume during concurrent I-frame transmission can be avoided.

[0045] In one possible implementation, the I-frame position is the frame number of the I-frame; the third device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, including: the third device using the frame number of the first frame image in the first image stream as the frame number of the first I-frame of the first image stream, and using the frame number of the first frame image in the second image stream as the frame number of the first I-frame of the second image stream; the third device determining the I-frame position weight of the first image stream and the I-frame position weight of the second image stream based on the first resolution and the second resolution; the third device determining the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate, where n is greater than or equal to 2.

[0046] In one possible implementation, the third device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate. This includes: the third device determining the frame number of the second I-frame of the first image stream; the third device determining the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate; the third device determining the frame number of the mth I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3; and the third device determining the frame number of the mth I-frame of the second image stream based on the frame number of the second I-frame of the second image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3.

[0047] In one possible implementation, the third device determines the I-frame position weights of the first image stream and the second image stream based on the first resolution and the second resolution, including: the third device determines the I-frame position weights of the first image stream using the following formula:

[0048]

[0049] Wherein, W1 is the I-frame position weight of the first image stream, S1 is the first resolution, and S2 is the second resolution;

[0050] The third device determines the I-frame position weights of the second image stream using the following formula:

[0051]

[0052] Wherein, W2 is the I-frame position weight of the second image stream, S1 is the first resolution, and S2 is the second resolution.

[0053] In one possible implementation, the third device determines the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate. This includes the third device determining the frame number of the second I-frame of the second image stream using the following formula:

[0054] P2 = P1 + (W1 + W2) * INR * Fps / 2

[0055] Wherein, P2 is the frame number of the second I-frame of the second image stream, P1 is the frame number of the second I-frame of the first image stream, W1 is the I-frame position weight of the first image stream, W2 is the I-frame position weight of the second image stream, INR is the I-frame time interval, and Fps is the frame rate.

[0056] In one possible implementation, the third device determines the frame number of the m-th I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: the third device determines the frame number of the m-th I-frame of the first image stream using the following formula:

[0057] P m =P m-1 +INR*Fps

[0058] Among them, P m P is the frame number of the m-th I-frame in the first image stream. m-1 INR is the frame number of the (m-1)th I-frame of the first image stream, Fps is the frame rate;

[0059] The third device determines the frame number of the m-th I-frame of the second image stream based on the frame number of the second I-frame, the I-frame time interval, and the frame rate, including: the third device determines the frame number of the m-th I-frame of the second image stream using the following formula:

[0060] P′ m =P′ m-1 +INR*Fps

[0061] Among them, P′ m Let P' be the frame number of the m-th I-frame in the second image stream. m-1 INR is the frame number of the (m-1)th I-frame of the second image stream, Fps is the frame rate.

[0062] In one possible implementation, before the third device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, the method further includes: the third device determining the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, wherein the first bandwidth threshold is greater than the second bandwidth threshold.

[0063] In one possible implementation, the third device determines the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, including: if the current network bandwidth is greater than or equal to the first bandwidth threshold, the third device uses a first preset interval as the I-frame time interval; if the current network bandwidth is greater than the second bandwidth threshold and less than the first bandwidth threshold, the third device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, wherein the second preset interval is greater than the first preset interval.

[0064] In one possible implementation, the third device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, including: the third device determines the I-frame time interval using the following formula:

[0065] INR = INR normal +(INR max -INR normal )*(BW-TH2) / (TH1-TH2)

[0066] Wherein, INR is the I-frame time interval, INR normal For the first preset interval, INR max The second preset interval is defined as BW, the current network bandwidth is defined as TH1, the first bandwidth threshold is defined as TH2, and the second bandwidth threshold is defined as TH2.

[0067] The encoding method provided in the second aspect refers to the first device as remote device 1 in this application embodiment, the second device as remote device 1 in this application embodiment, and the third device as local device in this application embodiment.

[0068] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the processor is configured to be coupled to the memory, read and execute instructions in the memory, so that the electronic device implements the steps executed by the first device in the encoding method provided in the first aspect.

[0069] Fourthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the processor is configured to be coupled to the memory, read and execute instructions in the memory, such that the electronic device implements the steps executed by the second device in the encoding method provided in the first aspect.

[0070] Fifthly, embodiments of this application provide a communication system, including: an electronic device provided in the third aspect and an electronic device provided in the fourth aspect.

[0071] In a sixth aspect, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the processor is configured to be coupled to the memory, read and execute instructions in the memory, such that the electronic device implements the steps performed by the first device in the encoding method provided in the second aspect.

[0072] In a seventh aspect, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the processor is configured to be coupled to the memory, read and execute instructions in the memory, such that the electronic device implements the steps performed by the second device in the encoding method provided in the second aspect.

[0073] Eighthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the processor is configured to be coupled to the memory, read and execute instructions in the memory, such that the electronic device implements the steps executed by the third device in the encoding method provided in the second aspect.

[0074] Ninthly, embodiments of this application provide a communication system, including: the electronic device provided in the sixth aspect, the electronic device provided in the seventh aspect, and the electronic device provided in the eighth aspect.

[0075] In a tenth aspect, embodiments of this application provide a readable storage medium storing a computer program. When executed, the computer program implements the steps executed by a first device in the encoding method provided in the first aspect, or implements the steps executed by a second device in the encoding method provided in the first aspect, or implements the steps executed by a first device in the encoding method provided in the second aspect, or implements the steps executed by a second device in the encoding method provided in the second aspect, or implements the steps executed by a third device in the encoding method provided in the second aspect.

[0076] Eleventhly, embodiments of this application provide a computer program product, wherein when the instructions contained in the computer program product are executed on a computer, the computer implements the steps executed by the first device in the encoding method provided in the first aspect, or implements the steps executed by the second device in the encoding method provided in the first aspect, or implements the steps executed by the first device in the encoding method provided in the second aspect, or implements the steps executed by the second device in the encoding method provided in the second aspect, or implements the steps executed by the third device in the encoding method provided in the second aspect.

[0077] The encoding method, electronic device, communication system, storage medium, and program product provided in this application compress multiple image streams. The resulting compressed streams have a much smaller data size than the corresponding image streams. The transmission time from the compressed streams to the local device is also shorter. Upon receiving the compressed streams, the local device can decode them to obtain the corresponding image streams, thus solving the latency problem caused by the long transmission time of the multiple image streams. Furthermore, before compressing the multiple image streams, the I-frame positions of the multiple image streams are calculated based on the resolution. For each image stream, an I-frame is generated at the corresponding position, ensuring an even distribution of I-frames across the multiple image streams. This avoids the image jitter problem caused by a sudden increase in data volume when I-frames are concurrently transmitted. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of an image sequence provided in an embodiment of this application;

[0079] Figure 2 This is a schematic diagram of the compression flow provided in an embodiment of this application;

[0080] Figure 3 Application scenarios provided for the embodiments of this application Figure 1 ;

[0081] Figure 4 Application scenarios provided for the embodiments of this application Figure 2 ;

[0082] Figure 5 Provided for the embodiments of this application Figure 3 The diagram illustrates concurrent I-frame operations in the scenario shown.

[0083] Figure 6 Provided for the embodiments of this application Figure 4 The diagram illustrates concurrent I-frame operations in the scenario shown.

[0084] Figure 7 A software framework diagram provided for embodiments of this application;

[0085] Figure 8 Provided for the embodiments of this application Figure 3 Interaction in the scene shown Figure 1 ;

[0086] Figure 9 A flowchart for determining the frame number of the second I-frame provided in an embodiment of this application;

[0087] Figure 10 Provided for the embodiments of this application Figure 3 I-frame distribution diagram in the scene shown;

[0088] Figure 11 Provided for the embodiments of this application Figure 3Interaction in the scene shown Figure 2 ;

[0089] Figure 12 Provided for the embodiments of this application Figure 3 Interaction in the scene shown Figure 3 ;

[0090] Figure 13 Provided for the embodiments of this application Figure 4 Interaction diagram of the scenario shown;

[0091] Figure 14 Provided for the embodiments of this application Figure 4 I-frame distribution diagram in the scene shown;

[0092] Figure 15 Provided for the embodiments of this application Figure 3 Software framework in the scenario shown Figure 1 ;

[0093] Figure 16 Provided for the embodiments of this application Figure 3 Software framework in the scenario shown Figure 2 ;

[0094] Figure 17 Provided for the embodiments of this application Figure 4 Software framework diagram for the scenario shown;

[0095] Figure 18 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application;

[0096] Figure 19 A software framework diagram of the electronic device 100 provided in the embodiments of this application. Detailed Implementation

[0097] To facilitate understanding of the solutions in the embodiments of this application by those skilled in the art, some terms used in the embodiments of this application are explained below:

[0098] Image Stream: The image sequence output by the Image Signal Processing (ISP) module after processing the image stream captured by the camera. The frame number of the first frame captured by the camera is denoted as P(1), the frame number of the second frame as P(2), ..., and the frame number of the nth frame as P(n). See also... Figure 1 As shown, the frame numbers of the image sequence output by the ISP module are consistent with the frame numbers of the images captured by the camera, namely P(1), P(2), ..., P(n). The ISP module can output multiple image streams after processing the image streams captured by multiple cameras.

[0099] Compressed Streams: Image streams contain a large amount of data. To facilitate storage or network transmission, image streams need to be compressed. A compressed stream can be understood as a sequence of frames obtained by compressing each frame of the image stream. When compressing images, intra-frame compression algorithms produce I-frames. I-frames carry all the information needed for decoding, therefore their data size is relatively large. Inter-frame compression algorithms produce P-frames. P-frames carry the differences between the current frame and the previous frame, and their data size is smaller than that of I-frames. Figure 2 It shows the Figure 1 The image stream shown is compressed to obtain the compressed stream.

[0100] Timestamp: The time when the camera captures an image. Assume the time when the camera captures the first frame is T(1), the time when it captures the second frame is T(2), ..., the time when it captures the nth frame is T(n). Then, the timestamp of the first frame in the image sequence output by the ISP module is T(1), the timestamp of the second frame is T(2), the timestamp of the third frame is T(3), ..., the timestamp of the nth frame is T(n).

[0101] Local devices: Devices that require the assistance of other devices to obtain multiple image streams. Local devices can take many forms, including but not limited to: mobile phones, tablets, laptops, or televisions.

[0102] Remote device: A device used to send image streams to a local device. Remote devices can take the form of, but are not limited to, mobile phones, tablets, laptops, or drones. The local and remote devices can connect via Bluetooth, Wi-Fi, 4G, or 5G technologies.

[0103] Figure 3 Application scenarios provided for the embodiments of this application Figure 1 , Figure 3 The communication system shown includes a local device and a remote device. The remote device includes multiple cameras that can simultaneously capture images. The local device can send shooting commands to the remote device via the network. After receiving the shooting command, the remote device controls the multiple cameras to capture images and sends the multiple image streams output by the ISP module to the local device via the network. The local device can then display these multiple image streams for user viewing, thus achieving the goal of obtaining multiple image streams from the remote device using the multiple cameras of the remote device. Figure 3The example shown uses a laptop computer as the local device and a mobile phone as the remote device, but this does not constitute a limitation on the embodiments of this application. Multiple cameras on the remote device may include a wide-angle camera, a main camera, and a telephoto camera.

[0104] Figure 4 Application scenarios provided for the embodiments of this application Figure 2 , Figure 4 The communication system shown includes a local device and at least two remote devices. Figure 4 The following example illustrates the use of at least two remote devices, including remote device 1 and remote device 2. The local device can send shooting commands to both remote device 1 and remote device 2 via the network. Upon receiving the shooting command, remote device 1 controls itself to acquire an image and sends the image stream output from its ISP module to the local device via the network. Similarly, upon receiving the shooting command, remote device 2 controls itself to acquire an image and sends the image stream output from its ISP module to the local device via the network. This allows the local device to simultaneously display the image streams sent by both remote devices. Figure 4 The example shown uses a laptop computer as the local device, a mobile phone as the remote device 1, and a drone as the remote device 2. This does not constitute a limitation on the embodiments of this application.

[0105] for Figure 3 In the scenario shown, because the data volume of each image stream output by the ISP module is very large, and network bandwidth is limited, the remote device needs to compress each image stream. Similarly, for Figure 4 In the scenario shown, remote device 1 and remote device 2 also need to compress the image stream output by the ISP module. For details, see [link to documentation]. Figure 5 As shown, in Figure 3 In the scenario shown, when compressing the image streams from the wide-angle camera, main camera, and telephoto camera, the first frame of each image stream needs to be encoded as an I-frame to enable successful decoding by the local device. With equal I-frame intervals, subsequent I-frames in the three compressed streams occur simultaneously. The large data volume of each I-frame causes a sudden increase in network data transmission, resulting in screen flickering on the local device and negatively impacting the user experience. Similarly, see... Figure 6 As shown, in Figure 4 In the scenario shown, when remote device 1 compresses the image stream output by the ISP module, it needs to encode the first frame of the image stream as an I-frame. When remote device 2 compresses the image stream output by the ISP module, it also needs to encode the first frame of the image stream as an I-frame. With the I-frame interval being the same, subsequent I-frames in the two compressed streams will occur simultaneously, which will also cause the amount of data transmitted over the network to increase instantaneously, causing the display screen of the local device to jitter.

[0106] In some embodiments, for Figure 3The image shaking issue described above can be resolved in the following ways:

[0107] See Figure 7 As shown, an encoding module, a transmission buffer, and a data transmission module are set up in the remote device. The encoding module is used to encode each image stream to obtain a compressed stream of each image stream, and stores the compressed stream of each image stream in the transmission buffer. Figure 7 Three compressed streams are shown, denoted as Stream 11, Stream 21, and Stream 31, respectively. The data transmission module is used to determine whether the image frames to be transmitted in each compressed stream are all I-frames. If so, the I-frames in each compressed stream are transmitted sequentially. For example, I-frame I11 from Stream 11 is transmitted first, followed by I-frame I21 from Stream 21, and then I-frame I31 from Stream 31. This avoids the problem of a sudden increase in the amount of data transmitted over the network when I-frames from each image stream are transmitted concurrently.

[0108] The local device is configured with a data receiving module, a decoding module, a receiving buffer, and a stable frame rate module. The data receiving module is used to receive the compressed streams sent by the data sending module. The decoding module is used to decode the compressed streams to obtain the decoded image streams and store the decoded image streams in the receiving buffer. Figure 7 In this diagram, stream 12 represents the image stream obtained after decoding stream 11, stream 22 represents the image stream obtained after decoding stream 21, and stream 32 represents the image stream obtained after decoding stream 31. The stable frame rate module retrieves the images to be output from the receive buffer and sends them to the data consumption module, thus ensuring that each image stream is transmitted to the data consumption module at the same frame interval. Because the data sending module does not simultaneously send I-frames from each compressed stream when I-frames are concurrently transmitted in the remote device, the arrival times of the I-frames from each compressed stream at the decoding module differ significantly. Figure 7 In this context, I12 represents the image obtained after decoding I11, I22 represents the image obtained after decoding I21, and I32 represents the image obtained after decoding I31. The frame rate stabilization module needs to wait a long time to obtain the I12, I22, and I32 to be output. The output of the frame rate stabilization module to the data consumption module is delayed, which increases the latency from the remote device to the local device and degrades the user's real-time experience.

[0109] This application provides an encoding / decoding method. To avoid concurrent I-frames of various image streams, it proposes to calculate the I-frame position of each image stream based on resolution, and generate an I-frame at the corresponding position for each image stream, so that the I-frames of each image stream are evenly distributed, thus avoiding concurrent I-frames of various image streams. At the same time, compared with the above embodiments, this method does not increase the latency from the remote device to the local device, thereby improving the user experience.

[0110] Applied to Figure 3 The scenarios shown can be categorized into three types based on the local device's control of the remote device:

[0111] Scenario 1: The local device does not control any camera on the remote device to capture images.

[0112] The second scenario: Based on the fact that the local device has already controlled the first and second cameras on the remote device to capture images, it is necessary to control the third camera on the remote device to also capture images.

[0113] The third scenario: The local device has already controlled the first and second cameras on the remote device to capture images, and the user wants to change the resolution of one of the image streams.

[0114] For the first scenario, the user can control the first and second cameras on the remote device to capture images by triggering the first command. The interaction process between the local and remote devices is described in [link to documentation]. Figure 8 As shown, it specifically includes:

[0115] S801, The local device sends a first instruction to the remote device. The first instruction may carry a first resolution and a second resolution. The first instruction is used to adjust the resolution parameter value corresponding to the first camera in the ISP module, and also to adjust the resolution parameter value corresponding to the second camera in the ISP module.

[0116] S802, the remote device configures the first resolution and the second resolution according to the first instruction.

[0117] The remote device can adjust the resolution parameter value corresponding to the first camera in the ISP module to the first resolution, and adjust the resolution parameter value corresponding to the second camera in the ISP module to the second resolution.

[0118] S803, remote device controls the first and second cameras to acquire images.

[0119] S804. The remote device processes the image stream captured by the first camera to obtain a first image stream, and processes the image stream captured by the second camera to obtain a second image stream.

[0120] The ISP module can process the image streams captured by the first camera and the image streams captured by the second camera respectively. Since the resolution parameter value corresponding to the first camera in the ISP module is adjusted to the first resolution in S802, the resolution of the first image stream processed by the ISP module is the first resolution. In S802, the resolution parameter value corresponding to the second camera in the ISP module is adjusted to the second resolution, so the resolution of the second image stream processed by the ISP is the second resolution.

[0121] For example, the first resolution is 2048×1080, and the second resolution is 1920×1080. The first camera is, for example, a wide-angle camera, and the second camera is, for example, a main camera. When the remote device receives the first instruction, it adjusts the resolution parameter value corresponding to the wide-angle camera in the ISP module to 2048×1080 and configures the resolution parameter value of the main camera to 1920×1080. The ISP module processes the image streams captured by the wide-angle camera and the image streams captured by the main camera, respectively. After adjusting the resolution parameter values ​​as described above, the resolution of the first image stream is 2048×1080, and the resolution of the second image stream is 1920×1080.

[0122] It should be noted that S803 can be executed before S802, or S803 can be executed simultaneously with S802. This application embodiment does not limit the execution order of the two steps.

[0123] The first and second resolutions in the first instruction above can be specified by the user.

[0124] S805. The remote device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate.

[0125] The I-frame position mentioned above can be represented by either the frame number or the timestamp.

[0126] The following describes the possible ways to determine the frame number of an I-frame:

[0127] To facilitate local device decoding, the frame number P(1) of the first frame in the first image stream can be used as the frame number of the first I-frame in the first image stream. Similarly, the frame number P(1) of the first frame in the second image stream can be used as the frame number of the first I-frame in the second image stream.

[0128] Since the first frame in each compressed stream is an I-frame, as described above, the concurrent display of multiple I-frames will cause the local device's screen to jitter. In order to reduce the user's perception of the jitter in the first frame, for each image stream, the local device decodes the I-frame and buffers the decoded image. After decoding the subsequent multiple images, the first frame image is then displayed, so that the first frame image displayed by the local device is quickly covered by the subsequent images, thereby reducing the user's perception of the jitter.

[0129] It is possible Figure 9 The method shown determines the frame number of the second I-frame. See also... Figure 9 As shown, it specifically includes:

[0130] S805-1. Determine the I-frame position weights of the first image stream and the second image stream based on the first resolution and the second resolution.

[0131] The I-frame position weights can be calculated using the following formula:

[0132]

[0133] Among them, W i S represents the position weight of frame I in the i-th image stream. i S is the resolution of the i-th image stream. j Let be the resolution of the j-th image stream.

[0134] Using the above formula, the I-frame position weights of the first image stream can be obtained as follows:

[0135]

[0136] Where W1 is the I-frame position weight of the first image stream, S1 is the first resolution, and S2 is the second resolution.

[0137] The I-frame position weights of the second image stream are:

[0138]

[0139] Where W2 is the I-frame position weight of the second image stream, S1 is the first resolution, and S2 is the second resolution.

[0140] The following example illustrates this:

[0141] The first resolution is, for example, 2048×1080, and the second resolution is, for example, 1920×1080. Using the above formula, the I-frame position weights of the first image stream can be obtained as follows:

[0142]

[0143] The I-frame position weights of the second image stream are:

[0144]

[0145] S805-2. Based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate, determine the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream, where n is greater than or equal to 2.

[0146] First, determine the frame number of the second I-frame of the first image stream.

[0147] Then, based on the I-frame position weights of the first image stream, the I-frame position weights of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, the frame number of the second I-frame of the second image stream is determined.

[0148] The frame number of the second I-frame of the i-th image stream can be calculated using the following formula:

[0149] P i =P i-1 +(W i-1 +W i )*INR*Fps / 2

[0150] Among them, P i W represents the frame number of the second I-frame in the i-th image stream. i P represents the I-frame position weight of the i-th image stream. i-1 W represents the frame number of the second I-frame in the (i-1)th video stream. i-1 This represents the I-frame position weight of the (i-1)th image stream. INR is the I-frame time interval, and Fps is the frame rate.

[0151] Using the above formula, the frame number of the second I-frame in the second image stream can be obtained as follows:

[0152] P2 = P1 + (W1 + W2) * INR * Fps / 2

[0153] Wherein, P2 is the frame number of the second I-frame of the second image stream, P1 is the frame number of the second I-frame of the first image stream, W1 is the I-frame position weight of the first image stream, and W2 is the I-frame position weight of the second image stream.

[0154] INR and Fps can be default values; for example, INR can be 1 second and Fps can be 30 frames / second.

[0155] INR can also be obtained in the following ways:

[0156] The I-frame time interval is determined based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, wherein the first bandwidth threshold is greater than the second bandwidth threshold.

[0157] In one possible implementation, the remote device can monitor the network bandwidth BW in real time and compare the network bandwidth BW with a first bandwidth threshold TH1 and a second bandwidth threshold TH2. When the network bandwidth BW is greater than or equal to TH1, the I-frame time interval is set to a first preset interval to ensure image quality. When the network bandwidth BW is less than or equal to TH2, it is considered that the current network quality cannot support the transmission of multiple image streams, and a prompt message is displayed to remind the user that the network bandwidth is insufficient. When the network bandwidth BW is greater than TH2 and less than TH1, the I-frame time interval is determined based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, wherein the second preset interval is greater than the first preset interval.

[0158] In one possible implementation, the I-frame time interval is calculated using the following formula 1.

[0159] INR = INR normal +(INR max -INR normal )*(BW-TH2) / (TH1-TH2) (Formula 1)

[0160] Wherein, INR represents the I-frame time interval, INR normal Indicates the first preset interval, INR max This represents the maximum acceptable I-frame time interval, also referred to as the second preset interval in this embodiment.

[0161] The following is about Figure 9 The method shown is illustrated with an example:

[0162] Assuming the frame number of the second I-frame in the first image stream is P(5), i.e., P1 = P(5), INR is 1s, Fps is 30 frames / s, and S10 calculates the I-frame position weight of the first image stream to be 0.52, and the I-frame position weight of the second image stream to be 0.48, then the frame number P2 of the second I-frame in the second image stream is:

[0163] P2=P(5+(0.52+0.48)*1*30 / 2)≈P(20)

[0164] See Figure 10 As shown, the second I-frame of the first image stream and the second image stream are evenly distributed, avoiding concurrency of I-frames.

[0165] pass Figure 9After obtaining the frame number of the second I-frame of the first image stream and the frame number of the second I-frame of the second image stream using the method shown, the frame number of the m-th I-frame of the first image stream is determined based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate. Similarly, the frame number of the m-th I-frame of the second image stream is determined based on the frame number of the second I-frame of the second image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3.

[0166] In one possible implementation, the frame number of the m-th I-frame of the first image stream can be determined by the following formula:

[0167] P m =P m-1 +INR*Fps

[0168] Among them, P m P is the frame number of the m-th I-frame in the first image stream. m-1 INR is the frame number of the (m-1)th I-frame in the first image stream, Fps is the frame rate, INR is the I-frame time interval, and Fps is the frame rate.

[0169] The frame number of the m-th I-frame in the second image stream can be determined using the following formula:

[0170] P′ m =P′ m-1 +INR*Fps

[0171] Among them, P′ m Let P′ be the frame number of the m-th I-frame in the second image stream. m-1 INR is the frame number of the (m-1)th I-frame in the second image stream, Fps is the frame rate, INR is the I-frame time interval, and Fps is the frame rate.

[0172] The following example illustrates this:

[0173] Assuming INR is 1s and Fps is 30 frames / s, if the frame number of the second I-frame in the first image stream is P(5), then the frame number of the third I-frame in the first image stream is P(5+1*30)=P(35). If the frame number of the second I-frame in the second image stream is P(20), then the frame number of the third I-frame in the image stream corresponding to the main camera is P(20+1*30)=P(50).

[0174] It should be noted that: See Figure 10 As shown, since the second I-frame of each image stream is evenly distributed and the interval between I-frames of each image stream is the same, the subsequent I-frames of each image stream are also evenly distributed.

[0175] The following describes the possible ways to determine the timestamp of an I-frame:

[0176] First, based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, the frame sequence number of the I-frame in the first image stream and the frame sequence number of the I-frame in the second image stream are determined. The specific process is described above and will not be repeated here in the embodiments of this application. Then, based on the frame sequence number of the I-frame in the first image stream, the timestamp of the first frame in the first image stream, and the frame rate, the timestamp of the I-frame in the second image stream is determined. Similarly, based on the frame sequence number of the I-frame in the second image stream, the timestamp of the first frame in the second image stream, and the frame rate, the timestamp of the I-frame in the second image stream is determined.

[0177] In one possible implementation, taking the first image stream as an example, the timestamp of the I-frame can be determined using Formula 2:

[0178] T i =T+(P i -1)*1 / Fps (Formula 2)

[0179] Among them, P i T represents the frame number of the I-frame in the first image stream. i The timestamp of the I-frame in the first image stream is represented by T, the timestamp of the first frame in the first image stream is represented by Fps, and the frame rate is represented by Fps.

[0180] S806. The remote device encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream, and encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream.

[0181] In one possible implementation, based on the frame number of the I-frame in the first image stream, it is determined whether the frame number of the image to be encoded in the first image stream is the same as the frame number of the I-frame. If so, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame. Similarly, it is determined whether the frame number of the image to be encoded in the second image stream is the same as the frame number of the I-frame in the second image stream. If so, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame.

[0182] The following example illustrates this:

[0183] The frame number of the second I-frame of the first image stream is obtained by the above method as P(5). It is then determined whether the frame number of the image to be encoded in the first image stream is P(5). If so, the image is encoded as an I-frame.

[0184] In another possible implementation, based on the timestamp of the I-frame of the first image stream, it is determined whether the timestamp of the image to be encoded in the first image stream is the same as the timestamp of the I-frame. If so, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame. Similarly, it is determined whether the timestamp of the image to be encoded in the second image stream is the same as the timestamp of the I-frame of the second image stream. If so, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame.

[0185] The following example illustrates this:

[0186] The timestamp of the second I-frame of the first image stream is obtained as T12 using the above method. It is then determined whether the timestamp of the image to be encoded in the first image stream is T12. If so, the image is encoded as an I-frame.

[0187] S807, The remote device sends the first compressed stream and the second compressed stream to the local device.

[0188] After receiving the compressed streams from each image stream, the local device decodes them to obtain the decoded image streams. The transmission time between the two compressed streams and the local device is short, resolving the latency issue caused by long transmission times of multiple image streams. Furthermore, the I-frames of each image stream are not concurrent, avoiding the image jitter problem caused by concurrent I-frames. Moreover, and... Figure 5 Compared to the illustrated embodiment, this method does not increase the latency between the remote device and the local device, thus improving the user experience.

[0189] For the second scenario, the local device has already passed... Figure 8 In the illustrated embodiment, based on controlling the first and second cameras to capture images, if the local device needs to control the third camera on the remote device to also capture images, the user can trigger a second command. The interaction process between the local device and the remote device is described in [reference needed]. Figure 11 As shown, it specifically includes:

[0190] S1101. The local device sends a second instruction to the remote device. The second instruction can carry a third resolution and is used to adjust the resolution parameter value corresponding to the third camera in the ISP module.

[0191] S1102. The remote device configures the third resolution according to the second instruction.

[0192] The remote device can adjust the resolution parameter value of the third camera in the ISP module to the third resolution.

[0193] S1103, Remote device controls the third camera to acquire images.

[0194] S1104. The remote device processes the image stream captured by the third camera to obtain the third image stream.

[0195] The image stream acquired by the third camera can be processed by the ISP module. Since the resolution parameter value corresponding to the third camera in the ISP module is adjusted to the third resolution in S1102, the resolution of the third image stream obtained by the ISP processing is the third resolution.

[0196] For example, the third resolution is 1280×720, and the third camera is a telephoto camera. After receiving the second instruction, the remote device adjusts the resolution parameter value corresponding to the telephoto camera in the ISP module to 1280×720. After this adjustment, the resolution of the third image stream processed by the ISP module is 1280×720.

[0197] It should be noted that S1103 can be executed before S1102, or S1103 can be executed simultaneously with S1102. This application embodiment does not limit the execution order of the two steps.

[0198] S1105. The remote device determines the I-frame position of the first image stream, the I-frame position of the second image stream, and the I-frame position of the third image stream based on the first resolution, the second resolution, the third resolution, the I-frame time interval, and the frame rate.

[0199] Because a third image stream has been added, the I-frame positions of each image stream need to be redefined. The process of redefined I-frame positions for each image stream is described below:

[0200] As described above, the position of an I-frame can be represented by either a frame number or a timestamp. The following describes the possible methods for re-determining the frame number of an I-frame:

[0201] First, when the third camera captures the first frame image, the frame numbers of the images captured by the first and second cameras can be updated to the frame number of the first frame image. Then, the frame number of the first frame image in the first image stream is used as the frame number of the first I-frame of the first image stream, the frame number of the first frame image in the second image stream is used as the frame number of the first I-frame of the second image stream, and the frame number of the first frame image in the third image stream is used as the frame number of the first I-frame of the third image stream. Then, based on the first resolution, the second resolution, and the third resolution, the I-frame position weights of the first, second, and third image streams are determined. Finally, based on the I-frame position weights of the first, second, and third image streams, the I-frame time interval, and the frame rate, the frame numbers of the nth I-frame of the first, second, and third image streams, respectively, are determined, where n is greater than or equal to 2.

[0202] The following example illustrates this:

[0203] Assuming the first camera is a wide-angle camera, the second camera is a main camera, and the third camera is a telephoto camera, and assuming that when the telephoto camera captures the first frame of the image, the wide-angle camera and the main camera capture the 100th frame of the image, the frame number of the 100th frame of the image can be updated to P(1), and then the frame number of the I-frame of each image stream can be determined using the method in S805.

[0204] The following describes the possible ways to redetermine the timestamp of an I-frame:

[0205] When the third camera captures the first frame image, the timestamps of the images captured by the first and second cameras can be updated to T(1), and then the timestamp of the I-frame of each image stream can be determined using the method in S805. See the description above for details; the embodiments of this application will not be repeated here.

[0206] S1106. The remote device encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream, encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream, and encodes the third image stream according to the I-frame position of the third image stream to obtain a third compressed stream.

[0207] The process of encoding each image stream by the remote device can be found in [reference needed]. Figure 8 The embodiment shown is S806, which will not be described again in this application.

[0208] S1107. The remote device sends the first compressed stream, the second compressed stream, and the third compressed stream to the local device.

[0209] After receiving the compressed streams of each image stream, the local device decodes the compressed streams of each image stream to obtain the decoded image stream. With the above encoding and decoding method, the local device can control more cameras to capture images, and the remote device can use the method in S805 to redetermine the I-frame position of each image stream, thereby avoiding the image jitter problem caused by concurrent I-frames.

[0210] For the third scenario, the local device has already passed... Figure 8 Based on the control of the first and second cameras to capture images in the illustrated embodiment, the user can trigger a third command to change the resolution of one of the image streams. For the interaction process between the local device and the remote device, please refer to [link to relevant documentation]. Figure 12 As shown, specifically including

[0211] S1201. The local device sends a third instruction to the remote device. The third instruction can carry a fourth resolution and is used to adjust the resolution parameter value corresponding to the first camera in the ISP module.

[0212] S1202. The remote device configures the fourth resolution according to the third instruction.

[0213] The remote device can adjust the resolution parameter value corresponding to the first camera in the ISP module to the fourth resolution.

[0214] S1203. The remote device processes the image stream captured by the first camera to obtain the fourth image stream.

[0215] The image stream acquired by the first camera can be processed by the ISP module. Since the resolution parameter value corresponding to the first camera in the ISP module is adjusted to the fourth resolution in S1202, the resolution of the fourth image stream obtained by the ISP processing is the fourth resolution.

[0216] For example, the first resolution is 1920×1080, and the first camera is a wide-angle camera. When the user wants to change the resolution of the image stream corresponding to the wide-angle camera output by the ISP module, a third instruction can be triggered. The third instruction carries a fourth resolution, for example, 2048×1080. After receiving the third instruction, the remote device adjusts the resolution parameter value corresponding to the wide-angle camera in the ISP module to 2048×1080. After this adjustment, the resolution of the fourth image stream processed by the ISP module is 2048×1080.

[0217] S1204. The remote device determines the I-frame position of the second image stream and the I-frame position of the fourth image stream based on the second resolution, the fourth resolution, the I-frame time interval, and the frame rate.

[0218] Because the resolution of the image stream output by the ISP module corresponding to the first camera changes from the first resolution to the fourth resolution, it is necessary to redetermine the I-frame positions of each image stream. The process of redetermining the I-frame positions of each image stream is described below:

[0219] As described above, the position of an I-frame can be represented by either a frame number or a timestamp. The following describes the possible methods for re-determining the frame number of an I-frame:

[0220] First, the frame numbers of the images captured by the first and second cameras when the third instruction is received can be updated to the frame number of the first frame image. Then, the frame number of the first frame image in the second image stream is used as the frame number of the first I-frame of the second image stream, and the frame number of the first frame image in the fourth image stream is used as the frame number of the first I-frame of the fourth image stream. Then, the I-frame position weights of the second and fourth image streams are determined according to the second and fourth resolutions. Finally, the frame numbers of the nth I-frame of the second and fourth image streams are determined according to the I-frame position weights of the second and fourth image streams, the I-frame time interval, and the frame rate, where n is greater than or equal to 2.

[0221] The following example illustrates this:

[0222] Assuming the first camera is a wide-angle camera and the second camera is a main camera, when the remote device receives the third instruction, the wide-angle camera and the main camera are capturing the 100th frame image. The frame number of the 100th frame image can be updated to P(1), and then the frame number of the I-frame of each image stream can be determined using the method in S805.

[0223] The following describes the possible ways to redetermine the timestamp of an I-frame:

[0224] When the third instruction is received, the timestamps of the images captured by the first and second cameras can be updated to T(1), and then the timestamps of the I-frames of each image stream can be determined using the method in S805. See the description above for details; the embodiments of this application will not be repeated here.

[0225] S1205. The remote device encodes the second image stream according to the I-frame position of the second image stream to obtain the second compressed stream, and encodes the fourth image stream according to the I-frame position of the fourth image stream to obtain the fourth compressed stream.

[0226] The process of encoding each image stream by the remote device can be found in [reference needed]. Figure 8 The embodiment shown is S806, which will not be described again in this application.

[0227] S1206. The remote device sends the second and fourth compressed streams to the local device.

[0228] After receiving the compressed streams of each image stream, the local device decodes the compressed streams of each image stream to obtain the decoded image stream. With the above encoding and decoding method, the local device can control the remote device to reconfigure the resolution and use the method in S805 to determine the position of the I-frame, which can avoid the screen jitter problem caused by concurrent I-frames.

[0229] Applied to Figure 4In the scenario shown, the user can control remote device 1 and remote device 2 to acquire images by triggering the first command. The interaction process between the local device, remote device 1, and remote device 2 is described in [reference needed]. Figure 13 As shown, it specifically includes:

[0230] S1301, the local device sends the first instruction to remote device 1 and remote device 2 respectively.

[0231] The local device sends a first instruction to the remote device 1 carrying a first resolution, which is used to adjust the resolution parameter value in the remote device 1; the local device sends a first instruction to the remote device 2 carrying a second resolution, which is used to adjust the resolution parameter value in the remote device 2.

[0232] S1302. Remote device 1 configures a first resolution according to the first instruction, and remote device 2 configures a second resolution according to the first instruction.

[0233] Remote device 1 can adjust the resolution parameter value in the ISP module to the first resolution. Remote device 2 can adjust the resolution parameter value in the ISP module to the second resolution.

[0234] S1303, Remote device 1 controls remote device 2 to acquire images, and remote device 2 controls remote device 2 to acquire images.

[0235] S1304. Remote device 1 processes the image stream acquired by remote device 1 to obtain a first image stream, and remote device 2 processes the image stream acquired by remote device 2 to obtain a second image stream.

[0236] The image stream acquired by remote device 1 can be processed by the ISP module in remote device 1. Since remote device 1 adjusts the resolution parameter value in the ISP module to a first resolution in S1302, the resolution of the first image stream processed by the ISP module is the first resolution. The image stream acquired by remote device 2 can be processed by the ISP module in remote device 2. Since remote device 2 adjusts the resolution parameter value in the ISP module to a second resolution in S1302, the resolution of the second image stream processed by the ISP module is the second resolution.

[0237] For example, the first resolution is, for instance, 2048×1080, and the second resolution is, for instance, 1920×1080. After receiving the first instruction, remote device 1 adjusts the resolution parameter value in the ISP module to 2048×1080. After this adjustment, the resolution of the first image stream output by the ISP module is 2048×1080. After receiving the first instruction, remote device 2 adjusts the resolution parameter value in the ISP module to 1920×1080. After this adjustment, the resolution of the second image stream output by the ISP module is 1920×1080.

[0238] S1305. The local device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate.

[0239] For details on how to determine the I-frame position, please refer to [link / reference]. Figure 8 The S805 step in the illustrated embodiment will not be repeated here. It should be noted that... (See also...) Figure 14 As shown, the average distribution of I-frames of each image stream calculated by S805 can avoid screen jitter caused by concurrent I-frames.

[0240] S1306. The local device sends the I-frame position of the first image stream to the remote device 1 and sends the I-frame position of the second image stream to the remote device 2.

[0241] S1307. Remote device 1 encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream. Remote device 2 encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream.

[0242] In one possible implementation, if the I-frame received by remote device 1 has the frame number of an I-frame, it is determined whether the frame number of the image to be encoded is the same as that frame number. If they are the same, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame. Similarly, if the I-frame received by remote device 2 has the frame number of an I-frame, it is determined whether the frame number of the image to be encoded is the same as that frame number. If they are the same, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame.

[0243] The following example illustrates this:

[0244] Taking the second I-frame as an example, combined with Figure 4 In the scenario shown, remote device 1 receives the frame number of the second I-frame as P(5). It then determines whether the frame number of the image to be encoded is P(5). If yes, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame.

[0245] In another possible implementation, if the I-frame received by remote device 1 has an I-frame timestamp, it determines whether the timestamp of the image to be encoded is the same as that timestamp. If they are the same, the image is encoded as an I-frame; otherwise, it is encoded as a P-frame. Similarly, if the I-frame received by remote device 2 has an I-frame timestamp, it determines whether the timestamp of the image to be encoded is the same as that timestamp. If they are the same, the image is encoded as an I-frame; otherwise, it is encoded as a P-frame.

[0246] Taking the second I-frame as an example, combined with Figure 4 In the scenario shown, remote device 1 receives the timestamp of the second I-frame as T12. It then determines whether the timestamp of the image to be encoded is T12. If it is, the image is encoded as an I-frame; otherwise, the image is encoded as a P-frame.

[0247] S1308, Remote device 1 sends a first compressed stream to the local device, and remote device 2 sends a second compressed stream to the local device.

[0248] Because remote device 1 compresses the first image stream and remote device 2 compresses the second image stream, the data volume of the two compressed streams is much smaller than the data volume of the corresponding image streams. The transmission time of the two compressed streams to the local device is short. After receiving the two compressed streams, the local device can decode them to obtain the corresponding image streams, thus solving the latency problem caused by the long transmission time of multiple image streams. Furthermore, the local device calculates the I-frame positions of the two image streams based on the resolution and sends these positions to the corresponding devices. This allows remote device 1 to generate an I-frame at the corresponding position based on the received I-frame position, and remote device 2 to generate an I-frame at the corresponding position based on the received I-frame position. Since the I-frames of the two image streams are evenly distributed, the image jitter problem caused by the instantaneous increase in data volume during concurrent I-frame transmission can be avoided.

[0249] Applied to Figure 3 The scene shown Figure 15 Software framework provided for embodiments of this application Figure 1 See Figure 15 As shown, the remote device includes, but is not limited to: a remote camera agent 13, a camera control module 14, an I-frame position calculation module 15, an I-frame generation control module 16, an encoding module 17, and a data transmission module 18. The encoding module 17 includes multiple encoding units. Figure 13 The diagram uses encoding units 1-3 as an illustration. The remote device also includes multiple cameras. Figure 15 The use of cameras 1-3 is illustrated in the diagram.

[0250] Local devices include, but are not limited to: camera application 24, camera control module 25, buffer calculation module 26, buffer configuration module 27, data receiving module 28, decoding module 29, frame rate smoothing module 30, and data output module 31. Decoding module 29 includes multiple decoding units. Figure 15 The decoding units 1-3 are illustrated in the diagram. The frame rate smoothing module 30 includes multiple buffers and a synchronization output module 301. Figure 15 Multiple buffers are illustrated using buffers 1-3.

[0251] To obtain multiple image streams using multiple cameras on a remote device, the user can trigger a first command on the camera application 24. The camera application 24 sends the first command to the camera control module 25, which in turn sends it to the remote camera agent 13. The remote camera agent 13 further sends the first command to the camera control module 14. The camera control module 14 configures a first resolution and a second resolution based on the first command and controls the first and second cameras to acquire images. The first and second cameras can be any two of cameras 1-3. The ISP module processes the image stream acquired by the first camera to obtain a first image stream and processes the image stream acquired by the second camera to obtain a second image stream.

[0252] The camera application 24, camera control module 25, remote camera agent 13, and camera control module 14 are also used to transmit the second and third instructions. The transmission process is the same as the first instruction described above, and will not be repeated here in the embodiments of this application.

[0253] The I-frame position calculation module 15 can be used to determine the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, and send the I-frame position of each image stream to the I-frame generation control module 16. The I-frame generation control module 16 is used to send I-frame generation requests to each encoding unit in the encoding module 17. Each encoding unit in the encoding module 17 is used to encode the first image stream according to the I-frame position of the first image stream to obtain the first compressed stream, and to encode the second image stream according to the I-frame position of the second image stream to obtain the second compressed stream, and send each compressed stream to the data transmission module 18.

[0254] In the local device, the data receiving module 28 receives compressed streams from various image streams and sends them to the decoding module 29. Each decoding unit in the decoding module 29 decodes each compressed stream to obtain the decoded images and stores them in their respective buffers. The buffer calculation module 26 determines the buffer size based on the expected maximum jitter time and frame rate. For example, if the expected maximum jitter time is 100ms and the frame rate is 30 frames / s, the buffer size can be approximately 100ms / (1s / (30 frames / s)) ≈ the space occupied by 3 frames; similarly, if the expected maximum jitter time is 100ms and the frame rate is 60 frames / s, the buffer size can be approximately 100ms / (1s / (60 frames / s)) ≈ the space occupied by 6 frames. The buffer configuration module 27 configures buffers 1-3 in the frame rate smoothing module 30 according to the buffer size determined by the buffer calculation module 26. The synchronization output module 301 is used to acquire images from buffers 1-3 and send the acquired images to the data output module 31. The synchronization output module 301 can ensure that the decoded image streams are output at the same frame interval. The data output module 31 can send the image streams to the camera application 24, which can be used to display the image streams.

[0255] Applied to Figure 3 The scene shown Figure 16 Software framework provided for embodiments of this application Figure 2 ,and Figure 15 The architecture shown is different. Figure 16 The remote device does not have an I-frame position calculation module 15, but an I-frame position calculation module 32 is designed in the local device. The I-frame position calculation module 32 is used to determine the I-frame position of the first image stream and the I-frame position of the second image stream according to the first resolution, the second resolution, the I-frame time interval and the frame rate, and send the I-frame position of each image stream to the I-frame generation control module 16 in the remote device. Figure 16 The functions of other software modules and Figure 15 Similarly, the embodiments of this application will not be described again here.

[0256] Applied to Figure 4 The scene shown Figure 17 For a software framework diagram provided in the embodiments of this application, see [link to software framework diagram]. Figure 17 As shown, remote device 1 includes, but is not limited to: remote camera agent 13, camera control module 14, camera 15, I-frame generation control module 16, encoding module 17, and data transmission module 18. Remote device 2 includes, but is not limited to: remote camera agent 13', camera control module 14', camera 15', I-frame generation control module 16', encoding module 17', and data transmission module 18'.

[0257] Local devices include, but are not limited to: camera application 24, camera control module 25, buffer calculation module 26, buffer configuration module 27, data receiving module 28, decoding module 29, frame rate smoothing module 30, data output module 31, and I-frame position calculation module 32. The decoding module 29 includes multiple decoding units. Figure 14 The use of decoding units 1-2 is illustrated. The frame rate smoothing module 30 includes multiple buffers and a synchronization output module 301. Figure 14 Multiple buffers are illustrated using buffer 1-2.

[0258] To obtain multiple image streams using multiple remote devices, taking remote device 1 as an example, the user can trigger a first command on the camera application 24. The camera application 24 sends the first command to the camera control module 25, which in turn sends it to the remote camera agent 13. The remote camera agent 13 then sends the first command to the camera control module 14. The camera control module 14 configures a first resolution based on the first command and controls the remote device 1 to acquire images. The ISP module processes the image stream acquired by the remote device 1 to obtain the first image stream. The processing procedure for remote device 2 is similar and will not be described further in this embodiment.

[0259] The I-frame position calculation module 32 determines the I-frame positions of the first image stream and the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate. It then sends the I-frame position of the first image stream to the I-frame generation control module 16 via the camera control module 25 and the remote camera agent 13. The I-frame generation control module 16 sends an I-frame generation request to the encoding module 17. The encoding module 17 encodes the first image stream based on its I-frame position to obtain a first compressed stream and sends it to the data transmission module 18. Similarly, the camera control module 25 and the remote camera agent 13 send the I-frame position of the second image stream to the I-frame generation control module 16. The I-frame generation control module 16 sends an I-frame generation request to the encoding module 17. The encoding module 17 encodes the second image stream based on its I-frame position to obtain a second compressed stream and sends it to the data transmission module 18.

[0260] The local device's data receiving module 28 receives compressed streams sent by various remote devices and sends them to the decoding module 29. Each decoding unit in the decoding module 29 decodes each compressed stream to obtain decoded images, which are then stored in their respective buffers. The buffer calculation module 26 determines the buffer size based on the expected maximum jitter time and frame rate. The buffer configuration module 27 configures buffers 1-2 in the frame rate smoothing module 30 according to the buffer size determined by the buffer calculation module 26. The synchronization output module 301 acquires images from buffers 1-2 and sends them to the data output module 31. The synchronization output module 301 ensures that the decoded image streams are output at the same frame interval. The data output module 31 sends the image streams to the camera application 24, which displays the image streams.

[0261] In one possible implementation, the local device can display multiple image streams in a left-right split-screen manner or in a top-bottom split-screen manner. This application does not limit the way the local device displays multiple camera streams.

[0262] Figure 18 A schematic diagram of the electronic device 100 is shown. Both remote and local devices can be used. Figure 18 The schematic structure is shown below. The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0263] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0264] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0265] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0266] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0267] Figure 18 The electronic device shown is Figure 3 When using a remote device in the scenario shown, the video codec includes... Figure 15 or Figure 16 Encoding units 1-3, Figure 18 The electronic device shown is Figure 3 When using a local device in the scenario shown, the video codec includes Figure 15 or Figure 16 Decoding units 1-3. Figure 18 The electronic device shown is Figure 4 When the remote device 1 is in the scenario shown, the video codec includes Figure 17 Chinese encoding module 17, Figure 18 The electronic device shown is Figure 4 When the remote device 2 is in the scenario shown, the video codec includes Figure 17 Chinese encoding module 17'. Figure 18 The electronic device shown is Figure 4 When using a local device in the scenario shown, the video codec includes Figure 17 Decoding units 1-2.

[0268] Figure 18 The electronic device shown is Figure 3 When a remote device is used in the scenario shown, camera 193 includes... Figure 15 or Figure 16 Cameras 1-3 in the middle. Figure 18 The electronic device shown is Figure 4 When using a remote device in the scenario shown, the video codec includes... Figure 17 Camera 15 in the middle.

[0269] See Figure 19 As shown, in some embodiments, the software of the electronic device 100 can be divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime, and the system library. The layers communicate with each other through software interfaces.

[0270] The application layer can include a series of application packages.

[0271] like Figure 19 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0272] Figure 18 The electronic device shown is Figure 3 or Figure 4 When the remote device is in the scenario shown, the application package may also include a remote camera agent. Figure 18 The electronic device shown is Figure 3 or Figure 4 When using a local device in the scenario shown, the application package may also include a camera application.

[0273] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0274] like Figure 19 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0275] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0276] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0277] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0278] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0279] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0280] The notification manager allows applications to display notifications in the status bar. These can be used to convey informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog layers on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0281] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0282] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0283] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0284] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), and system server module (SurfaceFlinger).

[0285] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0286] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0287] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0288] A 2D graphics engine is a graphics engine for 2D drawing.

[0289] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0290] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An encoding method applied to a communication system, the communication system comprising: A first device and a second device, the first device comprising: a first camera and a second camera, characterized in that the method comprises: The second device sends a first instruction to the first device; The first device configures a first resolution and a second resolution according to the first instruction; The first device controls the first camera and the second camera to capture images; The first device processes the image stream captured by the first camera to obtain a first image stream, the resolution of which is the first resolution; The first device processes the image stream captured by the second camera to obtain a second image stream, the resolution of which is the second resolution; The first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate; wherein the nth I-frame of the first image stream and the second image stream are evenly distributed to avoid concurrent I-frames of the first image stream and the second image stream; n is greater than or equal to 2; The first device encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream; The first device encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream; The first device sends the first compressed stream and the second compressed stream to the second device.

2. The method according to claim 1, characterized in that, The I-frame position is the frame sequence number of the I-frame; The first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, including: The first device uses the frame number of the first frame in the first image stream as the frame number of the first I-frame in the first image stream, and uses the frame number of the first frame in the second image stream as the frame number of the first I-frame in the second image stream. The first device determines the I-frame position weights of the first image stream and the I-frame position weights of the second image stream based on the first resolution and the second resolution. The first device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate.

3. The method according to claim 2, characterized in that, The first device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate, including: The first device determines the frame number of the second I-frame of the first image stream; The first device determines the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate. The first device determines the frame number of the m-th I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3. The first device determines the frame number of the m-th I-frame of the second image stream based on the frame number of the second I-frame of the second image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3.

4. The method according to claim 2, characterized in that, The first device determines the I-frame position weights of the first image stream and the second image stream based on the first resolution and the second resolution, including: The first device determines the I-frame position weights of the first image stream using the following formula: Wherein, W1 is the I-frame position weight of the first image stream, S1 is the first resolution, and S2 is the second resolution; The first device determines the I-frame position weights of the second image stream using the following formula: Wherein, W2 is the I-frame position weight of the second image stream, S1 is the first resolution, and S2 is the second resolution.

5. The method according to claim 3, characterized in that, The first device determines the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: The first device determines the frame number of the second I-frame of the second image stream using the following formula: P2 = P1 + (W1 + W2) * INR * Fps / 2 Wherein, P2 is the frame number of the second I-frame of the second image stream, P1 is the frame number of the second I-frame of the first image stream, W1 is the I-frame position weight of the first image stream, W2 is the I-frame position weight of the second image stream, INR is the I-frame time interval, and Fps is the frame rate.

6. The method according to claim 3, characterized in that, The first device determines the frame number of the m-th I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: The first device determines the frame number of the m-th I-frame of the first image stream using the following formula: P m =P m-1 +INR*Fps Among them, P m P is the frame number of the m-th I-frame in the first image stream. m-1 INR is the frame number of the (m-1)th I-frame of the first image stream, Fps is the frame rate; The first device determines the frame number of the m-th I-frame of the second image stream based on the frame number of the second I-frame, the I-frame time interval, and the frame rate, including: The first device determines the frame number of the m-th I-frame of the second image stream using the following formula: P′ m =P′ m-1 +INR*Fps Among them, P′ m Let P' be the frame number of the m-th I-frame in the second image stream. m-1 INR is the frame number of the (m-1)th I-frame of the second image stream, Fps is the frame rate.

7. The method according to any one of claims 1-6, characterized in that, Before the first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, the method further includes: The first device determines the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, wherein the first bandwidth threshold is greater than the second bandwidth threshold.

8. The method according to claim 7, characterized in that, The first device determines the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, including: If the current network bandwidth is greater than or equal to the first bandwidth threshold, the first device uses the first preset interval as the I-frame time interval. If the current network bandwidth is greater than the second bandwidth threshold and less than the first bandwidth threshold, the first device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, wherein the second preset interval is greater than the first preset interval.

9. The method according to claim 8, characterized in that, The first device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, including: The first device determines the I-frame time interval using the following formula: INR=INR normal +(INR max -INR normal )*(BW-TH2) / (TH1-TH2) Wherein, INR is the I-frame time interval, INR normal For the first preset interval, INR max The second preset interval is defined as BW, the current network bandwidth is defined as TH1, the first bandwidth threshold is defined as TH2, and the second bandwidth threshold is defined as TH2.

10. The method according to claim 1, characterized in that, The I-frame position is the timestamp of the I-frame; The first device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, including: The first device determines the frame number of the I-frame of the first image stream and the frame number of the I-frame of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate. The first device determines the timestamp of the I-frame of the first image stream based on the frame sequence number of the I-frame of the first image stream, the timestamp of the first frame image in the first image stream, and the frame rate. The first device determines the timestamp of the I-frame of the second image stream based on the frame sequence number of the I-frame of the second image stream, the timestamp of the first frame image in the second image stream, and the frame rate.

11. The method according to any one of claims 1-6 and 8-10, wherein the first device further comprises: The third camera, characterized in that the method further includes: The second device sends a second instruction to the first device; The first device configures a third resolution according to the second instruction; The first device controls the third camera to capture images; The first device processes the image stream captured by the third camera to obtain a third image stream, the resolution of which is the third resolution; The first device determines the I-frame position of the first image stream, the I-frame position of the second image stream, and the I-frame position of the third image stream based on the first resolution, the second resolution, the third resolution, the I-frame time interval, and the frame rate.

12. The method according to claim 11, characterized in that, The first device determines the I-frame position of the first image stream, the I-frame position of the second image stream, and the I-frame position of the third image stream based on the first resolution, the second resolution, the third resolution, the I-frame time interval, and the frame rate, including: When the first device captures the first frame image by the third camera, the frame number of the images captured by the first camera and the second camera is updated to the frame number of the first frame image; The first device uses the frame number of the first frame image in the first image stream as the frame number of the first I-frame of the first image stream, uses the frame number of the first frame image in the second image stream as the frame number of the first I-frame of the second image stream, and uses the frame number of the first frame image in the third image stream as the frame number of the first I-frame of the third image stream. The first device determines the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, and the I-frame position weight of the third image stream based on the first resolution, the second resolution, and the third resolution. The first device determines the frame number of the nth I-frame of the first image stream, the frame number of the nth I-frame of the second image stream, and the frame number of the nth I-frame of the third image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame position weight of the third image stream, the I-frame time interval, and the frame rate.

13. The method according to any one of claims 1-6 and 8-10, characterized in that, The method further includes: The second device sends a third instruction to the first device; The first device configures a fourth resolution according to the third instruction; The first device processes the image stream captured by the first camera to obtain a fourth image stream, wherein the resolution of the fourth image stream is the fourth resolution; The first device determines the I-frame position of the second image stream and the I-frame position of the fourth image stream based on the second resolution, the fourth resolution, the I-frame time interval, and the frame rate.

14. The method according to claim 13, characterized in that, The first device determines the I-frame position of the second image stream and the I-frame position of the fourth image stream based on the second resolution, the fourth resolution, the I-frame time interval, and the frame rate, including: When the first device receives the third instruction, the frame number of the images captured by the first camera and the second camera is updated to the frame number of the first frame image; The first device uses the frame number of the first frame in the second image stream as the frame number of the first I-frame in the second image stream, and uses the frame number of the first frame in the fourth image stream as the frame number of the first I-frame in the fourth image stream. The first device determines the I-frame position weight of the second image stream and the I-frame position weight of the fourth image stream based on the second resolution and the fourth resolution. The first device determines the frame number of the nth I-frame of the second image stream and the frame number of the nth I-frame of the fourth image stream based on the I-frame position weight of the second image stream, the I-frame position weight of the fourth image stream, the I-frame time interval, and the frame rate.

15. An encoding method applied to a communication system, the communication system comprising: The first device, the second device, and the third device are characterized in that the method includes: The third device sends a first instruction to the first device and the second device respectively; The first device configures a first resolution according to the first instruction; controls the first device to acquire images; processes the image stream acquired by the first device to obtain a first image stream, wherein the resolution of the first image stream is the first resolution; The second device configures a second resolution according to the first instruction; controls the second device to acquire images; processes the image stream acquired by the second device to obtain a second image stream, the resolution of the second image stream being the second resolution; The third device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate; sends the I-frame position of the first image stream to the first device; and sends the I-frame position of the second image stream to the second device; wherein the nth I-frame of the first image stream and the second image stream are evenly distributed to avoid concurrent I-frames of the first image stream and the second image stream; n is greater than or equal to 2; The first device encodes the first image stream according to the I-frame position of the first image stream to obtain a first compressed stream; and sends the first compressed stream to the third device. The second device encodes the second image stream according to the I-frame position of the second image stream to obtain a second compressed stream; and sends the second compressed stream to the third device.

16. The method according to claim 15, characterized in that, The I-frame position is the frame sequence number of the I-frame; The third device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, including: The third device uses the frame number of the first frame in the first image stream as the frame number of the first I-frame in the first image stream, and uses the frame number of the first frame in the second image stream as the frame number of the first I-frame in the second image stream. The third device determines the I-frame position weight of the first image stream and the I-frame position weight of the second image stream based on the first resolution and the second resolution. The third device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate.

17. The method according to claim 16, characterized in that, The third device determines the frame number of the nth I-frame of the first image stream and the frame number of the nth I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the I-frame time interval, and the frame rate, including: The third device determines the frame number of the second I-frame of the first image stream; The third device determines the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate. The third device determines the frame number of the m-th I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3. The third device determines the frame number of the m-th I-frame of the second image stream based on the frame number of the second I-frame of the second image stream, the I-frame time interval, and the frame rate, where m is greater than or equal to 3.

18. The method according to claim 16, characterized in that, The third device determines the I-frame position weights of the first image stream and the second image stream based on the first resolution and the second resolution, including: The third device determines the I-frame position weights of the first image stream using the following formula: Wherein, W1 is the I-frame position weight of the first image stream, S1 is the first resolution, and S2 is the second resolution; The third device determines the I-frame position weights of the second image stream using the following formula: Wherein, W2 is the I-frame position weight of the second image stream, S1 is the first resolution, and S2 is the second resolution.

19. The method according to claim 17, characterized in that, The third device determines the frame number of the second I-frame of the second image stream based on the I-frame position weight of the first image stream, the I-frame position weight of the second image stream, the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: The third device determines the frame number of the second I-frame of the second image stream using the following formula: P2 = P1 + (W1 + W2) * INR * Fps / 2 Wherein, P2 is the frame number of the second I-frame of the second image stream, P1 is the frame number of the second I-frame of the first image stream, W1 is the I-frame position weight of the first image stream, W2 is the I-frame position weight of the second image stream, INR is the I-frame time interval, and Fps is the frame rate.

20. The method according to claim 17, characterized in that, The third device determines the frame number of the m-th I-frame of the first image stream based on the frame number of the second I-frame of the first image stream, the I-frame time interval, and the frame rate, including: The third device determines the frame number of the m-th I-frame of the first image stream using the following formula: P m =P m-1 +INR*Fps Among them, P m P is the frame number of the m-th I-frame in the first image stream. m-1 INR is the frame number of the (m-1)th I-frame of the first image stream, Fps is the frame rate; The third device determines the frame number of the m-th I-frame of the second image stream based on the frame number of the second I-frame of the second image stream, the I-frame time interval, and the frame rate, including: The third device determines the frame number of the m-th I-frame of the second image stream using the following formula: P′ m =P′ m-1 +INR*Fps Among them, P′ m Let P' be the frame number of the m-th I-frame in the second image stream. m-1 INR is the frame number of the (m-1)th I-frame of the second image stream, Fps is the frame rate.

21. The method according to any one of claims 15-20, characterized in that, Before the third device determines the I-frame position of the first image stream and the I-frame position of the second image stream based on the first resolution, the second resolution, the I-frame time interval, and the frame rate, the method further includes: The third device determines the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, wherein the first bandwidth threshold is greater than the second bandwidth threshold.

22. The method according to claim 21, characterized in that, The third device determines the I-frame time interval based on the current network bandwidth, a first bandwidth threshold, and a second bandwidth threshold, including: If the current network bandwidth is greater than or equal to the first bandwidth threshold, the third device will use the first preset interval as the I-frame time interval; If the current network bandwidth is greater than the second bandwidth threshold and less than the first bandwidth threshold, the third device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, wherein the second preset interval is greater than the first preset interval.

23. The method according to claim 22, characterized in that, The third device determines the I-frame time interval based on the current network bandwidth, the first bandwidth threshold, the second bandwidth threshold, the first preset interval, and the second preset interval, including: The third device determines the I-frame time interval using the following formula: INR=INR normal +(INR max -INR normal )*(BW-TH2) / (TH1-TH2) Wherein, INR is the I-frame time interval, INR normal For the first preset interval, INR max The second preset interval is defined as BW, the current network bandwidth is defined as TH1, the first bandwidth threshold is defined as TH2, and the second bandwidth threshold is defined as TH2.

24. An electronic device, comprising: A memory and a processor, characterized in that the processor is configured to be coupled to the memory, read and execute instructions in the memory, such that the electronic device performs the steps performed by the first device according to any one of claims 1-14.

25. A communication system, characterized in that, The device includes the electronic device and the second device as described in claim 24, wherein the second device is used to perform the steps performed by the second device in any one of claims 1-14.

26. An electronic device comprising: A memory and a processor, characterized in that the processor is configured to be coupled to the memory, read and execute instructions in the memory, such that the electronic device performs the steps performed by the third device as claimed in any one of claims 15-23.

27. A communication system, characterized in that, The device includes the electronic device as described in claim 26, a first device, and a second device; the first device is used to implement the steps performed by the first device in any one of claims 15-23, and the second device is used to implement the steps performed by the second device in any one of claims 15-23.

28. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed, implements the steps performed by the first device in any one of claims 1-14, or the steps performed by the third device in any one of claims 15-23.

29. A computer program product, characterized in that, When the instructions contained in the computer program product are executed on a computer, the computer causes the computer to perform the steps performed by the first device as claimed in any one of claims 1-14, or to perform the steps performed by the third device as claimed in any one of claims 15-23.

Images