Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: receiving a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence comprises a first image; the second image sequence comprises a second image; the image content of the second image is the same as the image content of the first area in the first image; the first transmission frame rate is smaller than the second transmission frame rate; generating a third image sequence according to the first image sequence and the second image sequence; the third image sequence comprises a third image; the third images are in one-to-one correspondence with the second images; the image content of the first area in the third image is from the second image; image contents of other areas except the first area in the third image are from the first image; and generating a video stream according to the third image sequence, and transmitting the video stream to the user.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Currently, live video services have been applied in various fields, such as medical treatment, concerts, sports events, etc. The basic process of live video is: the camera shoots the live scene, transmits the captured image sequence to the cloud server in real time, and the cloud server generates live video streams of multiple perspectives based on the image sequence, and sends the required live video streams to the user based on the user's perspective information. In this process, the camera needs to upload the captured image sequence to the cloud server in real time, which has the problem of occupying a large uplink bandwidth. Summary of the invention

[0003] The embodiments of the present disclosure provide an image processing method, device, electronic device and storage medium, which can reduce the uplink bandwidth occupied by real-time transmission of image sequences and save computer resources during live video broadcast.

[0004] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0005] Receiving a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate;

[0006] A third image sequence is generated according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image one by one; the image content of the first area in the third image comes from the second image; the image content of other areas in the third image except the first area comes from the first image;

[0007] A video stream is generated according to the third image sequence, and the video stream is transmitted to a user.

[0008] In a second aspect, the present disclosure provides an image processing method, including:

[0009] A first image sequence is transmitted based on a first transmission frame rate, and a second image sequence is transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

[0010] In a third aspect, an embodiment of the present disclosure provides an image processing device, which is applied to a cloud server, including:

[0011] A data receiving unit, configured to receive a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate;

[0012] an image generating unit, configured to generate a third image sequence according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image in a one-to-one manner; the image content of the first area in the third image comes from the second image; and the image content of other areas in the third image except the first area comes from the first image;

[0013] A video transmission unit is used to generate a video stream according to the third image sequence, and transmit the video stream to a user.

[0014] In a fourth aspect, an embodiment of the present disclosure provides an image processing device, which is applied to an image capturing system, including:

[0015] A data transmission unit, used for transmitting a first image sequence based on a first transmission frame rate, and transmitting a second image sequence based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

[0016] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect or the second aspect above.

[0017] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0018] In one or more embodiments of the present disclosure, a first image sequence based on a first transmission frame rate is received, and a second image sequence based on a second transmission frame rate is received, the first image sequence includes a first image, the second image sequence includes a second image, the image content of the second image is the same as the image content of the first area in the first image, the first transmission frame rate is less than the second transmission frame rate, and a video stream is transmitted to a user based on the first image sequence and the second image sequence. It can be seen that through this embodiment, the first image can be transmitted at a low frame rate, that is, the first transmission frame rate, and the second image can be transmitted at a high frame rate, that is, the second transmission frame rate, and the image content of the second image is the same as the image content of the first area in the first image, so that high frame rate transmission is achieved for part of the content in the first image, such as the live broadcast content that the user focuses on, and low frame rate transmission is achieved for the rest of the content in the first image, such as the live broadcast content that the user does not pay attention to, which effectively reduces the uplink bandwidth occupied by the real-time transmission image sequence, and saves computer resources during live video in the live video scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0020] Figure 1 A schematic diagram of an application scenario of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a first area provided in accordance with an embodiment of the present disclosure;

[0023] Figure 4 A flowchart of an image processing method provided by another embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of an image processing device provided by another embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0028] The embodiments of the present disclosure provide an image processing method that can reduce the uplink bandwidth occupied by real-time transmission of image sequences and save computer resources during live video broadcasting. First, it should be noted that in each embodiment of the present disclosure, the frame rate refers to FPS (Frame Per Second, the number of frames transmitted per second).

[0029] Figure 1 A schematic diagram of an application scenario of an image processing method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the scene includes a camera, a camera side server, a cloud server and a user's terminal device. Among them, the camera and the camera side server are both located at the live broadcast site, and together constitute an image capture system, so the camera side server can also be called a live broadcast local server. In one embodiment, the camera side server can be a mobile phone or other handheld terminal device, or the camera side server can be integrated into the camera in the form of a service module. The cloud server is a remote server. The user's terminal device can be a VR (Virtual Reality) device. Figure 1 In the embodiment, the camera can capture images of the live scene, the camera side server communicates with the camera and the cloud server respectively, and the camera side server and the cloud server cooperate to execute the image processing method of this embodiment. The cloud server also sends a video stream to the user's terminal device through the image processing method of this embodiment, and the video stream can be a live video stream.

[0030] In one embodiment, Figure 1In the embodiment, the camera captures a first image at a preset frame rate and sends it to a camera-side server. The camera-side server generates a second image based on the first image, and the image content of the second image is the same as the image content of the first area in the first image. The camera-side server generates a first image sequence based on the first image, the first image sequence includes the first image, and the number of first images contained in the first image sequence per second is equal to the size of the first transmission frame rate. The camera-side server generates a second image sequence based on the second image, the second image sequence includes the second image, and the number of second images contained in the second image sequence per second is equal to the size of the second transmission frame rate. The camera-side server also transmits the first image sequence to the cloud server based on the first transmission frame rate, and transmits the second image sequence to the cloud server based on the second transmission frame rate. The first transmission frame rate is less than the second transmission frame rate, and the second transmission frame rate is less than or equal to the preset frame rate at which the camera captures the first image.

[0031] The cloud server generates a third image sequence according to the first image sequence and the second image sequence, the third image sequence includes a third image, the third image corresponds to the second image one-to-one, the image content of the first area in the third image comes from the second image, and the image content of other areas of the third image except the first area comes from the first image. The cloud server also generates a video stream according to the third image sequence and transmits the video stream to the user.

[0032] It can be seen that through the joint cooperation of the camera side server and the cloud server, the first image can be transmitted to the cloud server at a low frame rate, that is, the first transmission frame rate, and the second image can be transmitted to the cloud server at a high frame rate, that is, the second transmission frame rate, and the image content of the second image is the same as the image content of the first area in the first image. Therefore, for part of the content in the first image, such as the live broadcast content that the user focuses on, high frame rate transmission is achieved, and for the rest of the content in the first image, such as the live broadcast content that the user does not pay attention to, low frame rate transmission is achieved, which effectively reduces the uplink bandwidth occupied by uploading the captured image sequence to the cloud server in real time, and saves computer resources during live video in the live video scenario.

[0033] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the process includes:

[0034] Step S202, receiving a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of the first area in the first image; the first transmission frame rate is less than the second transmission frame rate;

[0035] Step S204, generating a third image sequence according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image one by one; the image content of the first area in the third image comes from the second image; the image content of other areas in the third image except the first area comes from the first image;

[0036] Step S206: Generate a video stream according to the third image sequence, and transmit the video stream to the user.

[0037] In this embodiment, a first image sequence based on a first transmission frame rate is received, and a second image sequence based on a second transmission frame rate is received, the first image sequence includes a first image, the second image sequence includes a second image, the image content of the second image is the same as the image content of the first area in the first image, the first transmission frame rate is less than the second transmission frame rate, and a video stream is transmitted to the user based on the first image sequence and the second image sequence. It can be seen that through this embodiment, the first image can be transmitted at a low frame rate, that is, the first transmission frame rate, and the second image can be transmitted at a high frame rate, that is, the second transmission frame rate, and the image content of the second image is the same as the image content of the first area in the first image, so that high frame rate transmission is achieved for part of the content in the first image, such as the live broadcast content that the user focuses on, and low frame rate transmission is achieved for the rest of the content in the first image, such as the live broadcast content that the user does not pay attention to, which effectively reduces the uplink bandwidth occupied by the real-time transmission image sequence, and saves computer resources during live video in the live video scene.

[0038] Of course, the method in this embodiment can also be applied in a video on demand scenario, and can also achieve the effect of reducing the uplink bandwidth occupied by the real-time transmission image sequence and saving computer resources. In the video on demand scenario, the video stream generated according to the third image sequence can be a video on demand stream.

[0039] Below Figure 2 The method flow in the application is applied to the cloud server as an example, and is described in detail Figure 2 Of course, Figure 2 The method process is not limited to being executed by a cloud server, but can also be executed by other devices such as a local server or a local terminal device, which is not limited in this embodiment.

[0040] In the above step S202, the cloud server receives a first image sequence transmitted by the image capture system based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate. For example, the cloud server receives a first image sequence transmitted by a camera-side server in the image capture system based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate. The first image sequence includes a first image captured by the image capture system, for example, includes a first image captured by a camera, and the number of first images contained in the first image sequence per second is equal to the size of the first transmission frame rate. The second image sequence includes a second image, and the number of second images contained in the second image sequence per second is equal to the size of the second transmission frame rate. The image content of the second image is the same as the image content of the first area in the first image. The first transmission frame rate is less than the second transmission frame rate.

[0041] In one embodiment, during live broadcast, the camera captures a first image of the live broadcast scene at a preset frame rate and sends it to a camera side server. The camera side server crops the first area in the first image to obtain a second image, and the second image is used to represent the image content of the first area in the first image.

[0042] The camera-side server generates a first image sequence according to the first image, and the number of first images contained in the first image sequence per second is equal to the size of the first transmission frame rate. For example, the preset frame rate is 60fps, the first transmission frame rate is 1fps, and the camera-side server selects the first first image from the 60 first images captured per second, and the first image selected per second forms the first image sequence.

[0043] The camera-side server generates a second image sequence based on the second image, and the number of second images contained in the second image sequence per second is equal to the size of the second transmission frame rate. For example, the preset frame rate is 60fps, the second transmission frame rate is 30fps, the camera-side server generates 60 corresponding second images based on 60 first images captured per second, selects 30 second images at intervals from the 60 second images per second, and the 30 second images selected per second form the second image sequence.

[0044] Finally, the camera-side server transmits the first image sequence to the cloud server based on the first transmission frame rate, and transmits the second image sequence to the cloud server based on the second transmission frame rate, so that the cloud server pushes the live video stream to the user.

[0045] In one embodiment, the first area is a predetermined area that the user pays the most attention to when watching the live broadcast. Therefore, by transmitting the second image at a high frame rate and the first image at a low frame rate, it is possible to save the uplink bandwidth occupied by uploading the captured image sequence to the cloud server in real time, while ensuring that the user can see the most concerned picture and the user's live viewing experience.

[0046] Figure 3 A schematic diagram of a first region provided in an embodiment of the present disclosure, such as Figure 3 As shown, the live broadcast scene includes a sunflower and surrounding background images such as grass and white clouds. When watching the live broadcast, users generally pay attention to the area where the sunflower is located. Therefore, in the first image taken by the camera, the area containing the sunflower can be determined as the first area. By transmitting the picture of the sunflower at a high frame rate and the overall picture taken by the camera at a low frame rate, the uplink bandwidth occupied by uploading the captured image sequence to the cloud server in real time can be effectively reduced without affecting the user's live viewing experience.

[0047] In the above step S204, the cloud generates a third image sequence according to the first image sequence and the second image sequence. The third image sequence includes a third image, and the third image corresponds to the second image one by one, so the number of third images contained in the third image sequence per second is equal to the size of the second preset frame rate. The image content of the first area in the third image comes from the second image, and the image content of other areas of the third image except the first area comes from the first image.

[0048] In one embodiment, the cloud server generates a third image sequence according to the first image sequence and the second image sequence, including:

[0049] Based on the second transmission frame rate, the first image sequence is expanded, and the first image in the expanded first image sequence corresponds to the second image one by one;

[0050] Using the second image to replace the image content of the first area in the corresponding first image, the first image after replacement is the third image;

[0051] A third image sequence is generated based on the third image.

[0052] In this embodiment, after the cloud server receives the first image sequence and the second image sequence, since the number of first images contained in the first image sequence per second is equal to the size of the first transmission frame rate, the number of second images contained in the second image sequence per second is equal to the size of the second transmission frame rate, and the second transmission frame rate is greater than the first transmission frame rate, the cloud server first expands the first image sequence based on the second transmission frame rate, and the first image in the expanded first image sequence corresponds one-to-one to the second image, and the number of first images contained in the expanded first image sequence per second is equal to the size of the second transmission frame rate.

[0053] In one embodiment, the first image sequence is expanded by duplicating the first image in the first image sequence. For example, the first transmission frame rate is 1fps, and the first image sequence contains 1 first image per second, and the second transmission frame rate is 30fps, and the second image sequence contains 30 second images per second. Based on this, it can be determined that each first image corresponds to 30 second images, and each first image is duplicated into 30 first images, thereby obtaining an expanded first image sequence, and the expanded first image sequence contains 30 first images per second, so that the first image in the expanded first image sequence corresponds to the second image one by one.

[0054] Next, on the basis of the one-to-one correspondence between the first image and the second image, the cloud server replaces the image content of the first area in the corresponding first image with the second image, and the replaced first image is the third image. Finally, the cloud server combines the third images into a third image sequence. It can be seen that in the third image sequence, the third image corresponds to the second image one-to-one, and the number of third images contained in the third image sequence per second is equal to the size of the second preset frame rate.

[0055] It can be seen that through this embodiment, the third image sequence can be generated by replacing the image content of the first area, using the first image sequence transmitted at a low frame rate and the second image sequence transmitted at a high frame rate. In the live broadcast scene, the first transmission frame rate is relatively low, resulting in a low real-time reception of the first image by the cloud server, and the second transmission frame rate is relatively high, resulting in a high real-time reception of the second image by the cloud server. Since the image content of the first area in the third image comes from the second image, and the image content of other areas in the third image except the first area comes from the first image, the real-time of the image content of the first area in the third image is relatively high, and the real-time of the image content of other areas except the first area is relatively low. Therefore, the third image can be called a weak real-time image in the live broadcast scene.

[0056] In one embodiment, replacing the image content of the first area in the corresponding first image with the second image includes:

[0057] Determine a coordinate correspondence between pixel points in the second image and pixel points in the first area of ​​the first image according to the width and height of the first image, the width and height of the first area, the spherical angle range covered by the first area, and the coordinates of the center point of the user's video screen;

[0058] According to the coordinate correspondence, the pixel points in the second image are used to replace the pixel points in the first area of ​​the corresponding first image.

[0059] In this embodiment, the coordinate correspondence between the pixel points in the second image and the pixel points in the first area of ​​the first image can be determined by the following formulas (1)-(4) based on the width and height of the first image, the width and height of the first area, the spherical angle range covered by the first area, and the coordinates of the center point of the user's video screen.

[0060]

[0061]

[0062] m2=u2 / F w *C w -0.5 (0≤m2<C w ) (3)

[0063] n2=v2 / F h *C h -0.5 (0≤n2<C h ) (4)

[0064] In formulas (1)-(4), W and H are the width and height of the first image, respectively, and C w , C h are the width and height of the first region, F w 、F h are the spherical angle range covered by the first area, namely F w *F h , m1, n1 are the coordinates of the pixel points in the first area of ​​the first image, m2, n2 are the coordinates of the pixel points in the second image, u1, v1 are the longitude and latitude coordinates of the pixel points in the first area of ​​the first image on the unit sphere, and u2, v2 can be obtained by multiplying u1, v1 with the inverse of the rotation matrix R. The premise for the validity of formulas (3) and (4) is: (0≤u2<F w ) and (0≤v2<F h ).

[0065] The rotation matrix R can be calculated based on the coordinates (yaw, pitch, roll) of the center point of the video screen when the user is watching the live broadcast. The specific calculation formula is as follows:

[0066]

[0067] In formula (5), α, β, and γ represent yaw, pitch, and roll, respectively. The coordinates of the center point of the user's video screen are the coordinates of the center point of the screen of the user's VR device, yaw represents the yaw angle, pitch represents the pitch angle, and roll represents the roll angle. The user's viewing angle information can be represented by the coordinates of the center point of the user's video screen.

[0068] In this embodiment, when distributing a live video stream to each user, it is necessary to generate a corresponding third image sequence for each user, that is, to generate a corresponding third image for each user. In the process of generating the third image through the above formula, the video screen center point coordinates (yaw, pitch, roll) used are the (yaw, pitch, roll) of each user. By calculating the rotation matrix R of each user according to the (yaw, pitch, roll) of each user, and multiplying u1 and v1 by the inverse of the rotation matrix R, the image part to be viewed by the user in the third image of each user can be rotated to the center of the user's field of view, so that when the user watches their respective live video stream, the viewing angle of the live video stream matches the user's viewing angle, thereby improving the live viewing experience.

[0069] After the above coordinate correspondence is calculated, the pixel points in the second image may be used to replace the pixel points in the first area of ​​the corresponding first image.

[0070] It can be seen that through this embodiment, the coordinate correspondence between the pixel points in the second image and the pixel points in the first area of ​​the first image can be determined according to the width and height of the first image, the width and height of the first area, the spherical angle range covered by the first area and the coordinates of the center point of the user's video screen. According to the coordinate correspondence, the pixel points in the second image can be efficiently and quickly used to replace the corresponding pixel points in the first area of ​​the first image, thereby improving the efficiency of generating the third image.

[0071] In the above step S206, the cloud server generates a video stream according to the third image sequence and transmits the video stream to the user. The video stream can be a live video stream or a video-on-demand stream. In one example, the cloud server crops the third image in the third image sequence according to the user's viewing angle information, generates a VAM video stream consistent with the user's viewing angle, and sends the VAM video stream as a live video stream or a video-on-demand stream to the user's VR device. The transmission frame rate of the live video stream or the video-on-demand stream transmitted to the user is less than or equal to the second transmission frame rate.

[0072] VAM (Visible Area + Margin) refers to the video screen transmitted to the user. When the user's video playback device is a VR device, the field of view of the video playback device can be represented by the FOV (Field of View) of the VR device. The field of view corresponding to VAM is greater than the field of view of the video playback device. In the VAM screen, the field of view of the visible area is equal to the field of view of the video playback device, and the margin is the extra field of view of the VAM screen relative to the video playback device.

[0073] The VAM video stream is sent to the user's VR device as a live or on-demand video stream. When the user wears the VR device and watches the video with a small head movement, the VR device can directly use the margin area Margin in the video screen for local rendering to meet the user's small head movement. There is no need to obtain a new video stream from the cloud server, so that the delay of the picture update is controlled within a preset time length such as 25ms, avoiding the user from feeling screen freezes or even dizziness.

[0074] In one embodiment, the first transmission frame rate and the second transmission frame rate may be frame rates set by a user, or may be frame rates adaptively adjusted according to network conditions.

[0075] In one embodiment, considering that the first image is a complete image taken by the camera, the first transmission frame rate used for transmitting from the camera-side server to the cloud server is low, which easily causes the user to take a long time to see the changed image content or miss the changed image content after the image content of other areas other than the first area in the first image changes, thereby reducing the user's video viewing experience. Based on this, in one embodiment, the second image sequence also includes a fourth image, and the fourth image is used to represent the image content of the second area in the first image. The image content of the fourth image is the same as the image content of the second area in the first image, and the second area is located in areas other than the first area. The above method flow also includes:

[0076] determining whether to update the first image according to a similarity between image content of the fourth image and image content in the second region of the first image;

[0077] If so, the first image is updated.

[0078] In this embodiment, the second image sequence also includes a fourth image, and the fourth image is used to represent the image content of the second area in the first image. The image content of the fourth image is the same as the image content of the second area in the first image. The second area is located in areas other than the first area. For example, the second area can be a small image area close to the first area, and the transmission frame rate of the fourth image is also the second transmission frame rate. The similarity between the image content of the fourth image and the image content in the second area in the first image is determined. If the similarity is less than the similarity threshold, it is determined that the image content of areas other than the first area has changed significantly, and the cloud server determines that the first image needs to be updated. Otherwise, it is determined that the image content of areas other than the first area has not changed significantly, and the cloud server determines that the first image does not need to be updated.

[0079] After determining that the first image needs to be updated, the cloud server notifies the camera-side server in the image capture system to send the first image at the current moment. After sending the first image, the camera-side server continues to transmit the first image sequence to the cloud server at the first transmission frame rate. For example, the camera-side server sends a first image to the cloud server at 02 seconds. According to the first transmission frame rate, the camera-side server should continue to send the first image to the cloud server at 04 seconds. However, at 03 seconds, the cloud server determines that the image content of the fourth image received most recently is less similar to the image content in the second area of ​​the first image received at 02 seconds. Then, the cloud server notifies the camera-side server to transmit a first image at 03 seconds. In response to the notification, the camera-side server sends the first image to the cloud server at 03 seconds and continues to transmit the first image sequence at the first transmission frame rate.

[0080] It can be seen that through this embodiment, by sending the fourth image sequence, it is equivalent to expanding some pixels around the first area to represent the image content of other areas, so that it can judge whether to update the first image according to the similarity between the image content of the fourth image and the image content in the second area of ​​the first image, thereby avoiding the situation where the user takes a long time to see the changed image content or misses the changed image content after the image content of other areas other than the first area in the first image changes due to the low first transmission frame rate used for transmission from the camera side server to the cloud server, thereby improving the user's video viewing experience.

[0081] In one embodiment, the first area is the area that the user focuses on, which can be a pre-set area by the user, or can be determined based on the focus points of a large number of users when watching videos. Based on this, in one embodiment, after transmitting the video stream to the user, the above method flow also includes:

[0082] Update the coordinates of the center point of the first area in the first image according to the coordinates of the center point of the video screen when the user watches the video stream;

[0083] The first region is updated in the first image based on the updated coordinates of the center point of the first region.

[0084] In this embodiment, the coordinates of the center point of the video screen when multiple users watch the video stream are obtained. When the user watches the video stream through the VR device, the coordinates of the center point of the video screen of the VR device are the coordinates of the center point of the video screen when the user watches the video stream. The VR device can transmit the coordinates of the center point of the video screen of the VR device to the cloud server in real time. The coordinates of the center point of the video screen when the user watches the video stream change with the change of the user's perspective, which can represent the user's perspective information.

[0085] The center point coordinates of the first area in the first image are updated according to the same or similar video screen center point coordinates of multiple users. For example, when the video screen center point coordinates of a large number of users are concentrated on and around the coordinates (10, 10, 10), the center point coordinates of the first area in the first image are updated according to the coordinates (10, 10, 10). For example, the coordinates (10, 10, 10) are used as the center point coordinates of the first area in the updated first image.

[0086] Next, the first area is updated in the first image based on the updated center point coordinates of the first area. For example, the updated center point coordinates of the first area are sent to the image capturing system. The updated center point coordinates of the first area are used to update the first area in the first image. The image capturing system can update the first area in the first image based on the updated center point coordinates of the first area and the preset size of the first area.

[0087] It can be seen that through this embodiment, when a user is watching a video stream, the center point coordinates of the first area in the first image can be updated according to the center point coordinates of the video screen when the user is watching the video stream, and the updated center point coordinates of the first area are sent to the image capture system to update the first area, so that the updated first area is the image area that most users focus on.

[0088] Figure 4 A flowchart of an image processing method provided by another embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0089] Step S402, transmitting a first image sequence based on a first transmission frame rate, and transmitting a second image sequence based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of the first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

[0090] It can be seen that through this embodiment, the first image can be transmitted at a low frame rate, that is, the first transmission frame rate, and the second image can be transmitted at a high frame rate, that is, the second transmission frame rate, and the second image is used to represent the image content of the first area in the first image, so that for part of the content in the first image, such as the live broadcast content that the user focuses on, high frame rate transmission is achieved, and for the remaining content in the first image, such as the live broadcast content that the user does not pay attention to, low frame rate transmission is achieved, which effectively reduces the uplink bandwidth occupied by the real-time transmission image sequence, and saves computer resources during live video in the live video scenario.

[0091] Figure 4The method flow in the embodiment can be applied to an image capturing system and executed by the image capturing system. Of course, it can also be applied to other devices such as a local server or a remote server, which is not limited here. Figure 4 The method flow in the image capture system is applied as an example and described in detail. Figure 4 specific process.

[0092] In one embodiment, before transmitting the first image sequence based on the first transmission frame rate and transmitting the second image sequence based on the second transmission frame rate, the method flow further includes:

[0093] capturing a first image at a preset capturing frame rate;

[0094] determining a first region in a first image;

[0095] The first region is cropped in the first image to obtain a second image.

[0096] In this embodiment, during live broadcast, the camera captures a first image of the live broadcast scene at a preset frame rate and sends it to a camera side server. The camera side server determines a first area in the first image, crops the first area in the first image to obtain a second image, and the second image is used to represent the image content of the first area in the first image.

[0097] It can be seen that through this embodiment, the first image can be captured according to the preset shooting frame rate, the first area can be determined in the first image, and the first area can be cropped in the first image to obtain the second image, thereby efficiently and quickly generating the first image and the second image.

[0098] In one embodiment, the first area, as the area focused on by the user, may be an area preset by the user, or may be determined based on the focus points of a large number of users when watching videos, or may be determined based on changes in image content of adjacent first images. Based on this, in one embodiment, determining the first area in the first image includes:

[0099] A first region is determined in the first image according to a difference in image contents of a plurality of first images with adjacent time stamps.

[0100] In one embodiment, during live broadcast, the camera captures the first image of the live broadcast scene at a preset frame rate and sends it to the camera-side server. The camera-side server marks a timestamp on each first image, and the time sequence of the timestamps is consistent with the sequence of the received first images. The camera-side server determines the center point coordinates of the area with a large change in image content as the center point coordinates of the first area based on the difference in image content of multiple first images with adjacent timestamps, and determines the first area in the first image based on the center point coordinates of the first area and the preset size of the first area.

[0101] In one embodiment, the difference between the image contents of the plurality of first images with adjacent timestamps can be calculated by calculating the difference between the pixel values ​​of the corresponding pixels, thereby determining the region with a large change in image content and the coordinates of the center point of the region. Alternatively, the difference between the image contents of the plurality of first images with adjacent timestamps can be determined by a pre-trained deep learning model, and the region with a large change in image content and the coordinates of the center point of the region can be determined by a pre-trained deep learning model.

[0102] Of course, in one embodiment, the camera side server can also determine that the area with a larger change in image content is the first area based on the difference in image content of multiple first images with adjacent timestamps. At this time, the size of the first area is not a preset size, but a size adaptively determined according to the change in image content of the first image.

[0103] It can be seen that through this embodiment, the first area can be determined in the first image according to the difference in image content of multiple first images with adjacent timestamps, so that the first area is an area where the image content changes greatly, and the first area is the area that the user focuses on when watching the video.

[0104] In one embodiment, cropping the first area in the first image to obtain the second image includes:

[0105] Rotate the first image according to the coordinates of the center point of the first area, so that the coordinates of the center point of the first area in the rotated first image are located at the center of the rotated first image;

[0106] In the rotated first image, the first area is cropped to obtain a second image.

[0107] In this embodiment, a rotation matrix is ​​first calculated based on the coordinates of the center point of the first region, and the first image is rotated based on the rotation matrix. In the rotated first image, the coordinates of the center point of the first region are located at the center of the rotated first image, so that the first region is rotated to the exact center of the first image, thereby increasing the pixel density of the first region.

[0108] The coordinates of the center point of the first region can be expressed as (α, β, γ), where α, β, and γ represent yaw, pitch, and roll, respectively. The rotation matrix R can be calculated based on the coordinates of the center point of the first region by referring to the above formula (5).

[0109] Then, in the rotated first image, the first area is cropped according to the size of the first area to obtain the second image.

[0110] It can be seen that, through this embodiment, by rotating the first image and cropping the first area in the rotated first image, the first area can be rotated to the exact center of the first image, thereby improving the pixel density of the first area.

[0111] In one embodiment, after transmitting the first image sequence based on the first transmission frame rate and transmitting the second image sequence based on the second transmission frame rate, the method flow further includes:

[0112] Obtain the updated center point coordinates of the first area;

[0113] The first region is updated in the first image according to the updated coordinates of the center point of the first region.

[0114] In this embodiment, the cloud server can obtain the coordinates of the center point of the video screen when multiple users watch the video stream in the manner described above. When the user watches the video stream through the VR device, the coordinates of the center point of the video screen of the VR device are the coordinates of the center point of the video screen when the user watches the video stream. The VR device can transmit the coordinates of the center point of the video screen of the VR device to the cloud server in real time. The coordinates of the center point of the video screen when the user watches the video stream change with the change of the user's perspective, which can represent the user's perspective information.

[0115] The cloud server updates the center point coordinates of the first area in the first image based on the same or similar video screen center point coordinates of multiple users. For example, when the video screen center point coordinates of a large number of users are concentrated on and around the coordinates (10, 10, 10), the center point coordinates of the first area in the first image are updated based on the coordinates (10, 10, 10). For example, the coordinates (10, 10, 10) are used as the center point coordinates of the first area in the updated first image.

[0116] Next, the cloud server sends the updated center point coordinates of the first area to the image capturing system, and the image capturing system may update the first area in the first image according to the updated center point coordinates of the first area and the preset size of the first area.

[0117] It can be seen that through this embodiment, the first area can be updated in real time while the user is watching the video stream, so that the updated first area is the image area that most users focus on.

[0118] The above describes the specific process of the image processing method applied to the image sending end, such as the image shooting system. For the working process of the image receiving end, such as the cloud server, or the parts not introduced here, please refer to Figure 1 The introduction will not be repeated here.

[0119] In summary, an image processing method is provided, which can transmit the first image at a low frame rate, i.e., the first transmission frame rate, and transmit the second image at a high frame rate, i.e., the second transmission frame rate, while ensuring that the user can watch live broadcast or on-demand video from any viewing angle. Thus, for part of the content in the first image, such as the live broadcast content that the user focuses on, high frame rate transmission is achieved, and for the rest of the content in the first image, such as the live broadcast content that the user does not pay attention to, low frame rate transmission is achieved, which effectively reduces the uplink bandwidth occupied by the real-time transmission of the image sequence, and saves computer resources during the live video broadcast in the live video broadcast scenario. Moreover, it has been proved through experiments that the above image processing method is applied in the live broadcast scenario, which can greatly save the uplink bandwidth occupied by the image sequence uploaded to the cloud server in real time without reducing the quality of the live broadcast picture.

[0120] Figure 5 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the device comprises:

[0121] A data receiving unit 51 is configured to receive a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate;

[0122] an image generating unit 52, configured to generate a third image sequence according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image in a one-to-one manner; the image content of the first area in the third image comes from the second image; the image content of other areas in the third image except the first area comes from the first image;

[0123] The video transmission unit 53 is used to generate a video stream according to the third image sequence, and transmit the video stream to the user.

[0124] Optionally, the image generating unit 52 is specifically configured to:

[0125] Based on the second transmission frame rate, the first image sequence is expanded, and the first image in the expanded first image sequence corresponds to the second image one by one;

[0126] Using the second image to replace the image content of the first area in the corresponding first image, so that the first image after replacement is the third image;

[0127] The third image sequence is generated according to the third image.

[0128] Optionally, the image generating unit 52 is further specifically configured to:

[0129] Determine, according to the width and height of the first image, the width and height of the first area, the spherical angle range covered by the first area, and the coordinates of the center point of the video screen of the user, a coordinate correspondence between pixel points in the second image and pixel points in the first area of ​​the first image;

[0130] According to the coordinate correspondence, the pixel points in the second image are used to replace the corresponding pixel points in the first area of ​​the first image.

[0131] Optionally, the second image sequence further includes a fourth image; image content of the fourth image is the same as image content of a second region in the first image; the second region is located in a region other than the first region; and the device further includes a first updating unit, which is used to:

[0132] determining whether to update the first image according to a similarity between image content of the fourth image and image content in the second region of the first image;

[0133] If so, the first image is updated.

[0134] Optionally, the device further includes a second updating unit, configured to:

[0135] After transmitting the video stream to the user, updating the coordinates of the center point of the first area in the first image according to the coordinates of the center point of the video screen when the user watches the video stream;

[0136] The first region is updated in the first image based on the updated center point coordinates of the first region.

[0137] The image processing device in the embodiment of the present disclosure can implement each process of the above-mentioned image processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0138] Figure 6 A structural diagram of an image processing device provided by another embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the device comprises:

[0139] A data transmission unit 61 is used to transmit a first image sequence based on a first transmission frame rate, and to transmit a second image sequence based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

[0140] Optionally, the device further includes: a determining and cutting unit, configured to:

[0141] Before transmitting the first image sequence based on the first transmission frame rate and transmitting the second image sequence based on the second transmission frame rate, capturing the first image at a preset capturing frame rate;

[0142] determining the first region in the first image;

[0143] The first region is cropped in the first image to obtain the second image.

[0144] Optionally, the determining and cutting unit is specifically used for:

[0145] The first region is determined in the first image according to a difference in image content of a plurality of the first images with adjacent time stamps.

[0146] Optionally, the determining and cutting unit is specifically used for:

[0147] Rotate the first image according to the coordinates of the center point of the first area, so that the coordinates of the center point of the first area in the rotated first image are located at the center of the rotated first image;

[0148] In the rotated first image, the first area is cropped to obtain the second image.

[0149] Optionally, the device further includes a third updating unit, configured to:

[0150] After transmitting the first image sequence based on the first transmission frame rate and the second image sequence based on the second transmission frame rate, the updated center point coordinates of the first area are acquired; and the first area is updated in the first image according to the updated center point coordinates of the first area.

[0151] The image processing device in the embodiment of the present disclosure can implement each process of the above-mentioned image processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0152] An embodiment of the present disclosure further provides an electronic device, Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 7As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 701 and memory 702, and one or more applications or data may be stored in the memory 702. Among them, the memory 702 may be a temporary storage or a permanent storage. The application stored in the memory 702 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 701 may be configured to communicate with the memory 702 to execute a series of computer executable instructions in the memory 702 on the electronic device. The electronic device may also include one or more power supplies 703, one or more wired or wireless network interfaces 704, one or more input or output interfaces 705, one or more keyboards 706, etc.

[0153] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0154] Receiving a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate;

[0155] A third image sequence is generated according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image one by one; the image content of the first area in the third image comes from the second image; the image content of other areas in the third image except the first area comes from the first image;

[0156] A video stream is generated according to the third image sequence, and the video stream is transmitted to a user.

[0157] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned image processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0158] In another specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0159] A first image sequence is transmitted based on a first transmission frame rate, and a second image sequence is transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

[0160] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned image processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0161] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0162] Receiving a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate;

[0163] A third image sequence is generated according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image one by one; the image content of the first area in the third image comes from the second image; the image content of other areas in the third image except the first area comes from the first image;

[0164] A video stream is generated according to the third image sequence, and the video stream is transmitted to a user.

[0165] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned image processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0166] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0167] A first image sequence is transmitted based on a first transmission frame rate, and a second image sequence is transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

[0168] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned image processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0169] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0170] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0171] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0172] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0173] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0174] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0179] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0180] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0181] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. An image processing method, characterized in that: include: receiving a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; The first image sequence includes a first image; The second image sequence includes a second image; the image content of the second image is the same as the image content of the first area in the first image; the first transmission frame rate is lower than the second transmission frame rate; A third image sequence is generated according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image one by one; the image content of the first area in the third image comes from the second image; the image content of other areas in the third image except the first area comes from the first image; A video stream is generated according to the third image sequence, and the video stream is transmitted to a user.

2. The method according to claim 1, characterized in that The step of generating a third image sequence according to the first image sequence and the second image sequence comprises: Based on the second transmission frame rate, the first image sequence is expanded, and the first image in the expanded first image sequence corresponds to the second image one by one; Using the second image to replace the image content of the first area in the corresponding first image, so that the first image after replacement is the third image; The third image sequence is generated according to the third image.

3. The method according to claim 2, characterized in that The using the second image to replace the image content of the first area in the corresponding first image includes: Determine, according to the width and height of the first image, the width and height of the first area, the spherical angle range covered by the first area, and the coordinates of the center point of the video screen of the user, a coordinate correspondence between pixel points in the second image and pixel points in the first area of ​​the first image; According to the coordinate correspondence, the pixel points in the second image are used to replace the corresponding pixel points in the first area of ​​the first image.

4. The method according to claim 1, characterized in that: The second image sequence further includes a fourth image; the image content of the fourth image is the same as the image content of the second area in the first image; the second area is located in an area other than the first area; The method further comprises: determining whether to update the first image according to a similarity between image content of the fourth image and image content in the second region of the first image; If so, the first image is updated.

5. The method according to claim 1, characterized in that After transmitting the video stream to the user, the method further includes: updating the coordinates of the center point of the first area in the first image according to the coordinates of the center point of the video screen when the user watches the video stream; The first region is updated in the first image based on the updated center point coordinates of the first region.

6. An image processing method, characterized in that: include: transmitting a first image sequence based on a first transmission frame rate, and transmitting a second image sequence based on a second transmission frame rate; The first image sequence includes a first image; The second image sequence includes a second image; the image content of the second image is the same as the image content of the first area in the first image; and the first transmission frame rate is lower than the second transmission frame rate.

7. The method according to claim 6, characterized in that Before transmitting the first image sequence based on the first transmission frame rate and transmitting the second image sequence based on the second transmission frame rate, the method further includes: capturing the first image at a preset capturing frame rate; determining the first region in the first image; The first region is cropped in the first image to obtain the second image.

8. The method according to claim 7, characterized in that The determining the first area in the first image includes: The first region is determined in the first image according to a difference in image content of a plurality of the first images with adjacent time stamps.

9. The method according to claim 7, characterized in that: The step of cropping the first region in the first image to obtain the second image includes: Rotate the first image according to the coordinates of the center point of the first area, so that the coordinates of the center point of the first area in the rotated first image are located at the center of the rotated first image; In the rotated first image, the first area is cropped to obtain the second image.

10. The method according to claim 6, characterized in that After transmitting the first image sequence based on the first transmission frame rate, and transmitting the second image sequence based on the second transmission frame rate, the method further includes: Obtaining updated coordinates of the center point of the first area; The first region is updated in the first image according to the updated coordinates of the center point of the first region.

11. An image processing device, characterized in that: Applied to cloud servers, including: A data receiving unit, configured to receive a first image sequence transmitted based on a first transmission frame rate and a second image sequence transmitted based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate; an image generating unit, configured to generate a third image sequence according to the first image sequence and the second image sequence; the third image sequence includes a third image; the third image corresponds to the second image in a one-to-one manner; the image content of the first area in the third image comes from the second image; and the image content of other areas in the third image except the first area comes from the first image; A video transmission unit is used to generate a video stream according to the third image sequence, and transmit the video stream to a user.

12. An image processing device, characterized in that: Applied to image capture systems, including: A data transmission unit, used for transmitting a first image sequence based on a first transmission frame rate, and transmitting a second image sequence based on a second transmission frame rate; the first image sequence includes a first image; the second image sequence includes a second image; the image content of the second image is the same as the image content of a first area in the first image; the first transmission frame rate is less than the second transmission frame rate.

13. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, wherein the computer executable instructions, when executed, cause the processor to implement the steps of the method described in any one of claims 1 to 5, or implement the steps of the method described in any one of claims 6 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 above are implemented, or the steps of the method described in any one of claims 6 to 10 above are implemented.

Images