Video streaming method, device, equipment and storage medium based on panoramic image

Abstract

The present application relates to image processing technology, and provides a video diversion method, device, equipment and storage medium based on panoramic images. The method determines the size of the diverted image according to the resolution of the panoramic image, and sets the starting coordinates of the diverted image in the panoramic image. Then, according to the diverted image size and the preset overlapping image size, the starting coordinates of all diverted images in the panoramic image can be determined in sequence; the image data of the panoramic image is read through a row stride encoder, and according to the starting coordinates of each diverted image in the panoramic image and the diverted image size, the diverted image data can be directly output from the panoramic image data, thereby directly outputting the diverted image without copying, cropping and then splicing the panoramic image, reducing the memory pressure and data processing volume of the processing device, and improving the diversion output efficiency of the panoramic image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video streaming method, device, equipment and storage medium based on panoramic images. Background Art

[0002] Existing security systems use multiple surveillance devices to monitor the same area from different directions, thus achieving complete coverage. However, these multiple devices require corresponding processing equipment to process the images from each device, increasing system deployment costs. Consequently, existing security systems are gradually adopting panoramic cameras to cover the entire surveillance area.

[0003] However, the panoramic images output by panoramic cameras have high resolutions, and loading and outputting these high-resolution images requires powerful data processing capabilities. Existing surveillance equipment typically copies panoramic images in segments and then encodes these copied parts. This results in a high data processing load and low efficiency in the output of panoramic image streams. Therefore, addressing this low efficiency in the output of panoramic image streams has become a pressing technical challenge. Summary of the Invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for video diversion based on panoramic images, aiming to solve the technical problem of low efficiency of existing panoramic image diversion output.

[0005] To achieve the above-mentioned objectives, the present invention provides a video splitting method based on panoramic images, which includes: setting the splitting image size according to the effective resolution of the panoramic image; setting the starting coordinates of each splitting image according to the splitting image size, the preset overlapping area size and the preset coordinate position of the splitting image in the panoramic image; inputting the splitting image size, the row pixel value of the panoramic image and the starting coordinates of each splitting image into a row stride encoder to output the splitting image.

[0006] In addition, to achieve the above-mentioned purpose, the present invention also provides a video diversion device based on panoramic images, and the video diversion device based on panoramic images includes: a diversion image size setting module, which is used to set the diversion image size according to the effective resolution of the panoramic image; a starting coordinate setting module, which is used to set the starting coordinates of each diversion image according to the diversion image size, the preset overlapping area size and the preset coordinate position of the diversion image in the panoramic image; a diversion image output module, which is used to input the diversion image size, the row pixel value of the panoramic image and the starting coordinates of each diversion image into a row stride encoder to output the diversion image.

[0007] In addition, to achieve the above-mentioned purpose, the present invention also provides a panoramic image-based video diversion device, which includes a processor, a memory, and a panoramic image-based video diversion program stored on the memory and executable by the processor, wherein when the panoramic image-based video diversion program is executed by the processor, the steps of panoramic image-based video diversion as described above are implemented.

[0008] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a video diversion program based on panoramic images is stored, wherein when the video diversion program based on panoramic images is executed by a processor, the steps of the video diversion method based on panoramic images as described above are implemented.

[0009] The present invention provides a video splitting method based on a panoramic image. The method sets the split image size according to the effective resolution of the panoramic image; sets the starting coordinates of each split image according to the split image size, the preset overlapping area size, and the preset coordinate position of the split image in the panoramic image; and inputs the split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image into a row stride encoder to output the split image. Through the above-mentioned method, the present invention determines the size of the split image according to the resolution of the panoramic image, and sets the starting coordinates of the split image in the panoramic image. Then, according to the preset overlapping image, the starting coordinates of all the split images in the panoramic image can be determined in sequence; the image data of the panoramic image is read through the row stride encoder, and according to the starting coordinates of the split image and the size of the split image, the split image data can be directly output from the panoramic image data, thereby outputting the split image without copying, cropping, and splicing the panoramic image, thereby reducing the amount of data processing, improving the split output efficiency of the panoramic image, and solving the technical problem of low split output efficiency of the existing panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Schematic diagram of the hardware structure of a panoramic image-based video streaming device involved in an embodiment of the present invention;

[0011] Figure 2 1 is a flow chart of a first embodiment of a method for video splitting based on panoramic images according to the present invention;

[0012] Figure 3 This is a flow chart of a second embodiment of a method for video splitting based on panoramic images according to the present invention;

[0013] Figure 4 Schematic diagram of image splitting of a video splitting method based on panoramic images according to the present invention;

[0014] Figure 5 Schematic diagram of the flow of a third embodiment of a method for video offloading based on panoramic images according to the present invention;

[0015] Figure 6 1. A schematic flow chart of a fourth embodiment of a method for video splitting based on panoramic images according to the present invention;

[0016] Figure 7 Schematic diagram of functional modules of the first embodiment of the video splitting device based on panoramic images of the present invention.

[0017] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] The panoramic image-based video splitting method involved in the embodiment of the present invention is mainly applied to a panoramic image-based video splitting device, which can be a PC, portable computer, mobile terminal or other device with display and processing functions.

[0020] Reference Figure 1 , Figure 1 Schematic diagram of the hardware structure of the panoramic image-based video splitting device involved in the embodiment of the present invention. In the embodiment of the present invention, the panoramic image-based video splitting device may include a processor 1001 (such as a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface); the memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0021] Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation on the video streaming device based on panoramic images, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0022] Continue to refer to Figure 1 , Figure 1The memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, and a video streaming program based on panoramic images.

[0023] exist Figure 1 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the video diversion program based on panoramic images stored in the memory 1005, and execute the video diversion method based on panoramic images provided in an embodiment of the present invention.

[0024] An embodiment of the present invention provides a video streaming method based on panoramic images.

[0025] Reference Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a method for video splitting based on panoramic images according to the present invention.

[0026] In this embodiment, the panoramic image-based video streaming method includes the following steps:

[0027] Step S10, setting the size of the split image according to the effective resolution of the panoramic image;

[0028] In this embodiment, the panoramic image is captured by a panoramic camera, which uses at least two fisheye lenses to capture images and stitch them together into a 360-degree panoramic video stream. However, due to the wide-angle characteristics of the fisheye lens, the top and bottom of the captured image are significantly distorted and have no reference value. Therefore, it is necessary to crop the less distorted portion of the panoramic image to create the valid image. For example, if the panoramic image resolution is 7680×3840, the cropped effective resolution is 7680×1440, with a total of 2000 rows of pixel values ​​cropped from the top and bottom.

[0029] Specifically, due to the characteristics of the fisheye lens, the panoramic image is distorted only at the top and bottom, while the pixels in the middle row are not distorted. After cropping the distorted pixels at the top and bottom of the panoramic image, the pixel row values ​​of the panoramic image decrease, while the column values ​​do not change. Therefore, the effective resolution of the panoramic image is the product of the original column values ​​of the panoramic image and the cropped row values.

[0030] It can be understood that the effective row pixel value of the panoramic image refers to the column value of the effective resolution, that is, the number of pixels in a row of the effective resolution indicates the number of columns of the effective resolution; the effective column pixel value refers to the row value of the effective resolution, that is, the number of pixels in a column of the effective resolution indicates the number of rows of the effective resolution.

[0031] Furthermore, the step S10 specifically includes:

[0032] determining the row value of the split image resolution according to the effective resolution of the panoramic image;

[0033] According to a preset diversion angle, the proportion of the diversion image in the panoramic image is calculated, and the column value of the resolution of the diversion image is determined.

[0034] In this embodiment, in the effective resolution of the panoramic image, because pixels with large distortion in the panoramic image have been excluded before the effective resolution is determined, if the diverted image wants to cover the effective resolution, the row value (i.e., width) in the effective resolution is used as the resolution row value of the diverted image.

[0035] It can be understood that if the row value (i.e., width) of the effective resolution of the panoramic image is too large, the row value can also be divided, and multiple diversion images can be set to cover the effective resolution of the panoramic image; for example, if the effective resolution row value is 1000, the diversion image resolution row value can be set to 100, and there is a certain overlapping area between the two diversion images to ensure that the panoramic image can be fully covered to avoid image loss.

[0036] In this embodiment, a preset diversion angle is set based on the preset number of diverted videos and the 360-degree panoramic image. The overlapping coverage area needs to be considered here. According to the ratio of the preset diversion angle to 360 degrees, the area of ​​the panoramic image that can be covered by the preset diversion angle is calculated. The panoramic image resolution column value (i.e., length) contained in the coverage area is the column value of the diverted image resolution.

[0037] Furthermore, the size of the diversion resolution can be determined by combining the row value and the column value.

[0038] It can be understood that diversion is to intercept the corresponding angle of the picture from the panoramic 360-degree picture and send it to the receiving end.

[0039] In a specific embodiment, the angle is configured by the backend (or monitoring), and each stream can be set to any angle. For example, a panoramic video stream is split into four ordinary video streams (non-panoramic). The operating terminal can pull all four video streams simultaneously. The horizontal FOV of each video stream is 120 degrees. To ensure full coverage of the panoramic image, adjacent video streams have a 30-degree overlap. Taking a panoramic image of 7680×1440 as an example, after being split horizontally into four streams, the image resolution of each stream is 2560×1440, and the overlap between adjacent streams is 640×1440.

[0040] It can be understood that the angle of each of the four streams can be set individually. The camera can be configured by calling the interface on the camera side through the operation end. The camera will intercept the stream of the corresponding angle from the panoramic stream according to the configuration and send it to the monitoring.

[0041] Step S20, setting the starting coordinates of each diversion image according to the diversion image size, the preset overlapping area size, and the preset coordinate position of the diversion image in the panoramic image;

[0042] In this embodiment, after determining the size of the diverted image, an encoding starting coordinate needs to be given to the encoder so that the encoder can read the diverted image in the panoramic image; in order to cover all valid areas of the panoramic image and avoid the problem of missing images, a preset overlapping area of ​​a certain range is required between the diverted images. A reasonable overlapping area size can be set according to the diverted image size and the number of diversions.

[0043] Specifically, the first column of pixels of the panoramic image can be used as the starting column coordinates of the first diversion image, and the coordinates of the first row of pixels of the effective resolution of the panoramic image in the panoramic image can be used as the starting row coordinates of the first diversion image, so as to determine the starting coordinates of the first diversion image in the panoramic image, and then deduce the starting coordinates of other diversion images in the panoramic image based on the overlapping area; the starting coordinates of each diversion image can also be manually set according to the actual needs of the user.

[0044] It is understandable that the preset overlapping area size cannot be too large, otherwise it will easily lead to excessive image repetition, data redundancy, and waste of computing power; it cannot be too small, otherwise the coverage effect cannot be achieved. If the edge area of ​​the display device is not fully displayed, it will easily lead to omissions in the edge area of ​​the displayed image.

[0045] Specifically, the size of the overlapping area can be set according to reasonable rules, such as setting the overlapping area to account for 15% of the diversion image, or setting a certain angle range as the overlapping area when setting the angle of the diversion image.

[0046] Furthermore, the step S20 specifically includes:

[0047] Determining a readable area of ​​the panoramic image by the row stride encoder according to an effective resolution of the panoramic image;

[0048] Setting the starting coordinates of the diversion image in the panoramic image according to the readable area and the size of the diversion image;

[0049] The starting coordinates of adjacent diversion images of the diversion image are determined according to the preset overlapping image size and the starting coordinates.

[0050] In this embodiment, the row value (width) of the panoramic image resolution is first determined, and then the position of the effective resolution in the panoramic image resolution is determined. Because the column value (length) of the effective resolution is equal to the column value of the panoramic image resolution, it is only necessary to determine that the first row of pixels of the effective resolution is located at the row coordinates of the panoramic image resolution. The row coordinates of the first row of pixels of the effective image resolution are used as the starting row coordinates of the first diversion image.

[0051] Specifically, the row coordinates of the first row of pixels of the effective resolution in the panoramic image resolution are first determined. Then, the row coordinates of the last row of pixels of the effective resolution in the panoramic image resolution can be determined based on the row value (width) of the effective resolution. Combined with the size of the effective resolution, the coordinates of the row stride encoder in the readable area of ​​the panoramic image can be determined.

[0052] Optionally, the starting row coordinates of the split image resolution are set in the readable area according to the split image resolution; for example, the starting row coordinates of the effective resolution can be used as the starting row coordinates of the first split image resolution.

[0053] Specifically, the starting column coordinates of the diversion image resolution can be set through the operation end. The first column of the effective image resolution can be used as the starting column coordinates of the first diversion image resolution by default, or any column of the effective image resolution can be set as the starting column coordinates of the first diversion image resolution.

[0054] Specifically, the resolution coordinates of the first diversion image in the panoramic image can be determined based on the starting row coordinates and starting column coordinates of the resolution of the first diversion image; then, the resolution coordinates of each diversion image in the panoramic image can be determined in turn based on the preset overlapping resolution.

[0055] Step S30, inputting the size of the split image, the row pixel value of the panoramic image, and the starting coordinates of the split image into a row stride encoder to output the split image;

[0056] Generally speaking, to capture a small image from a large image, you usually need to apply for a block of memory from the system, then copy the pixels of the small image to this block of memory, and then give this block of memory to the encoder to encode and generate a video stream. Since panoramic videos are relatively large, this requires consuming more CPU and memory resources, which greatly affects performance on embedded systems.

[0057] In this embodiment, the application uses the encoder's row stride (memory image row span) to directly send the large image to the encoder. For example, the encoding area is set to 2560×1440 and the stride is 7680, eliminating the memory copy part to achieve the required performance. The encoder reads data according to the resolution of the large panoramic image, and reads the pixel data of the panoramic image in sequence according to the row order of the panoramic image resolution. For ease of understanding, each row of pixels of the panoramic image resolution is regarded as a group of data. Assuming that the panoramic image is 7680×1440, it means that the stride encoder will read 1440 groups of pixel data. If the starting coordinates of the first split image in the panoramic image are (0, 0), then the stride encoder actually needs to encode the first to 2560th data in each group of pixel data, and the data segment from the 2561st to the 7680th data of each row does not need to be processed or is directly deleted. That is, after the stride encoder completes the encoding of the first to 2560th data in the previous row of pixel data, the subsequent data of the previous row is no longer encoded, and the encoding continues from the first data of the next row to the 2560th data of the pixel data in that row, and the same processing is performed on each row of pixel data in sequence. Finally, the encoded pixel data is output in sequence, thereby outputting the first split image.

[0058] To facilitate understanding by those skilled in the art, this embodiment is explained by taking the generation process of the first diversion image as an example.

[0059] Specifically, the generation process of the first diversion image based on the panoramic image is as follows:

[0060] In a specific embodiment, a panoramic image with row pixel values, a split image resolution size (i.e., split image size), and a starting coordinate of the first split image to be captured are input into an encoder, and the encoder calls a memory image row stride plug-in to process the panoramic image.

[0061] Optionally, the encoder reads the row pixel data of the panoramic image in row order, determines the diverted image data to be encoded in the read panoramic image data according to the starting coordinates of the diverted image and the row value in the diverted image resolution size, processes the diverted image data by calling the row stride plug-in, and directly outputs the diverted image.

[0062] In this embodiment, the resolution size covered by the overlapping area can be calculated according to the preset diversion angle, and the starting column coordinates of the second diversion image in the panoramic image can be determined by calculation based on the edge pixel columns of the first diversion image.

[0063] Specifically, the resolution of each diversion image is the same. When the starting column of the second diversion image is determined, the ending column of the second diversion image can be determined; and by analogy, the starting column and ending column of each diversion image in the panoramic image can be determined, and the resolution coordinates of each diversion image in the panoramic image can be determined.

[0064] Specifically, the resolution coordinates of each branched image to be intercepted are sequentially input into the encoder, and the encoding start coordinates of each branched image are determined. Combined with the branched image size, the video stream of each branched image can be output sequentially.

[0065] In this embodiment, the resolution coordinates of each channel can be determined based on the preset overlapping area size and the starting coordinates of the first shunt image. The corresponding image data is read from the panoramic image through the row stride encoder and then output as shunt image data, thereby generating multiple channels of video in sequence.

[0066] In this embodiment, the data of the panoramic image is read through a row stride encoder, and then the image data corresponding to the diverted image is directly output according to the coordinates of the diverted image in the panoramic image, while the invalid data of the non-diverted image data is not processed or is directly deleted, which reduces memory pressure and data processing volume, thereby improving the output efficiency of the diverted image.

[0067] The present invention provides a video splitting method based on a panoramic image. The method sets the split image size according to the effective resolution of the panoramic image; sets the starting coordinates of the split image according to the split image size and the preset overlapping image size; and inputs the split image size, the row pixel value of the panoramic image, and the starting coordinates of the split image into a row stride encoder to output the split image. Through the above-mentioned method, the present invention determines the size of the split image according to the resolution of the panoramic image, and sets the starting coordinates of the split image in the panoramic image. Then, according to the preset overlapping image, the starting coordinates of all the split images in the panoramic image can be determined in sequence; the image data of the panoramic image is read through the row stride encoder, and according to the starting coordinates of the split image and the split image size, the split image data can be directly output from the panoramic image data, thereby outputting the split image without copying, cropping, and then splicing the panoramic image, thereby reducing the amount of data processing, improving the split output efficiency of the panoramic image, and solving the technical problem of low split output efficiency of the existing panoramic image.

[0068] Reference Figure 3 , Figure 3 FIG. 4 is a flow chart of a second embodiment of a method for video splitting based on panoramic images according to the present invention.

[0069] Based on the above Figure 2In the embodiment shown, in this embodiment, step S30 includes:

[0070] Step S31, running the row stride encoder according to the starting coordinates of the split image to read the image data of the panoramic image;

[0071] Step S32, determining valid data in the image data read by the row stride encoder according to the size of the split image and the row pixel values ​​of the panoramic image;

[0072] In this embodiment, the row alignment feature of stride is utilized to allow the encoder to output the data of the first 1 / 4 encoding area (2560×1440) in each data cycle in stride as the data of the shunted image, and the data of the last 3 / 4 is deleted as invalid data, so that the shunted image can be directly output without processing redundant data.

[0073] Furthermore, the step S32 specifically includes:

[0074] Reading the panoramic image row pixels in sequence through the row stride encoder to obtain panoramic image encoded data;

[0075] Determining, according to the starting coordinates of the diversion image and the size of the diversion image, valid row pixels of each row of the diversion image in a corresponding row of the panoramic image;

[0076] According to the valid row of pixels, valid data corresponding to the split image is determined in the panoramic image coded data.

[0077] In this embodiment, after distortion screening, although the effective image resolution of the panoramic image is determined, invalid pixels are not cropped. Therefore, it is necessary to determine the position of the effective resolution in the panoramic image, that is, the coordinates of the first row of pixels and the last row of pixels of the effective resolution in the panoramic image.

[0078] Specifically, the encoder operates on the entire panoramic image, while the row stride encoder directly "filters" the diverted image data from the panoramic image data. Therefore, to correctly and completely read the diverted image, it is necessary to determine the position of the diverted image in the panoramic image, that is, the starting coordinates of the diverted image.

[0079] Specifically, the effective resolution starting row can be determined to be located at the row coordinates of the panoramic image according to the distortion screening operation, and the starting column of the first branch image can be any column of the effective resolution, starting from the first column or other columns.

[0080] It can be understood that during the encoding process, the row stride encoder reads data in sequence according to the rows of the panoramic image. For example, the encoder reads the data of the first row of pixels of the full-view image before reading the data of the second row of pixels. The data length of each row of pixels is the same. The coordinates of the split image and the resolution of the split image "tell" the row stride encoder how to "distinguish" valid data from invalid data.

[0081] Specifically, assuming that the effective resolution of the panoramic image is 10000×5000, the starting interception coordinates of the shunt image are (1000, 1000), and the resolution of the shunt image is 3000×1000; then the row stride encoder will read all the effective resolution data of the panoramic image, which is arranged in row order according to the effective resolution of the panoramic image, that is, 5000 rows of data, and each row of data has 10,000 pixel data; assuming that these data are numbered, with 10,000 pixel data as a cycle data; when the shunt image data needs to be output, the row stride encoder will start from the 1000th cycle data, starting from the 1000th pixel data in each row of data, and read 3000 pixel data, a total of 1000 corresponding pixel data in the cycle data as valid data, that is, shunt image data, output the valid data, and then convert it into a shunt image; the other panoramic image data is regarded as invalid data and can be directly deleted.

[0082] Specifically, refer to Figure 4 , Figure 4 This is a schematic diagram of image splitting in the panoramic image-based video splitting method of the present invention. The stride encoder inputs the starting coordinates of each split image, the split image resolution (width and height), and the panoramic image width. The panoramic image is divided into multiple split regions corresponding to the split images. Adjacent split images have overlapping regions, ensuring that the output split images fully cover the panoramic image.

[0083] In a specific embodiment, the stride encoder reads pixel data for each row of the panoramic image in sequence. In the row corresponding to the row coordinates of the split image in the panoramic image, the encoder extracts the split image row corresponding to the split image width value from the corresponding row pixel data based on the starting coordinates of each split image. The encoder then skips the remaining rows of pixel data and continues reading the corresponding split image row in the next row of the panoramic image until all rows of pixel data for the split image are read.

[0084] Among them, when the shunted image crosses rows, since the stride encoder reads pixel data in sequence according to the row order, there is a row of pixel misalignment in the edge shunted image that crosses rows. At this time, you can choose to preprocess the panoramic image before shunting, and cut out the part of the panoramic image that crosses rows and splice it to the other side of the panoramic image, so as to obtain a panoramic image with a larger resolution and avoid the stride encoder from reading the edge shunted image across rows.

[0085] Step S33: Output the valid data through the row stride encoder as the shunt image data to obtain the shunt image.

[0086] In this embodiment, through the above implementation, there is no need to pre-process the panoramic image (such as segmentation, stitching, etc.). It is only necessary to directly read the corresponding pixel data from the panoramic image through the row stride encoder as the data of the diverted image, thereby directly outputting the diverted image. The data that needs to be processed is only the data corresponding to the diverted image, which reduces the amount of data processing and thus improves the output efficiency of the diverted image.

[0087] Reference Figure 5 , Figure 5 FIG. 4 is a flow chart of a third embodiment of a method for video offloading based on panoramic images according to the present invention.

[0088] Based on the above Figure 2 In the embodiment shown, in this embodiment, before step S10, the following steps are further included:

[0089] Step S01, screening all pixels of the panoramic image according to a preset distortion threshold, and determining an area where the distortion amount of the pixels is within the distortion threshold as a valid area;

[0090] Step S02: determining the effective resolution of the panoramic image according to the pixel values ​​contained in the effective area.

[0091] In this embodiment, a distortion threshold parameter is set, and distortion is measured for all pixels in the panoramic image to calculate the distortion amount of each pixel. Pixel rows where more than half of the pixels have a distortion amount greater than the distortion threshold are determined to be invalid pixel rows. Pixel rows where more than half of the pixels have a distortion amount less than the distortion threshold are determined to be valid pixel rows, and distortion correction is performed.

[0092] Specifically, distortion, also known as distortion, is an aberration that can turn straight lines into curves. Wide-angle lenses, zoom lenses, and telephoto lenses are prone to severe distortion. It's particularly noticeable along tangent lines at the edges of the image, but is invisible along radial lines. Due to the inherent characteristics of optical lenses, distortion (distortion) in captured images is detrimental to image quality.

[0093] Furthermore, after distortion correction, the pixel area determined to be a valid pixel row is regarded as a valid area, and all pixels contained in the area are valid pixels, thereby determining the effective resolution of the panoramic image.

[0094] Reference Figure 6 , Figure 6 FIG. 4 is a flow chart of a fourth embodiment of a method for video splitting based on panoramic images according to the present invention.

[0095] Based on the above Figure 2 In the embodiment shown, in this embodiment, before step S30, the following steps are further included:

[0096] Step S031, determining the missing image size of the edge diversion image according to the diversion image size, the preset overlapping image size and the starting coordinates of the diversion image;

[0097] Step S032, based on the size of the missing image, starting from the first column on the other side of the panoramic image, intercepting a supplementary image with the same size as the missing image;

[0098] Step S033 , aligning and splicing the supplementary image to the edge position of the panoramic image, and updating the panoramic image so that the edge branch image is read completely.

[0099] In this embodiment, because there is an overlapping area between the two adjacent shunt images, the edge shunt image on one side of the panoramic image is repeated with the shunt image on the other side of the panoramic image. The encoder reads the image pixel rows in sequence. After completing the reading of a row of pixel data, it will jump to the first pixel of the next row of pixels and continue reading backward. When the edge shunt image and the shunt image on the other side are repeated, when the encoder reads the data, the front and back data of each row of pixel data of the edge shunt image correspond to two rows of pixel data of the panoramic image, so each row of pixel data of the edge shunt image is misaligned by one row. Therefore, in order to read the edge shunt image completely and arrange the pixel data in row order, the panoramic image needs to be preprocessed before reading the shunt image; that is, the supplementary images are intercepted and spliced, and the panoramic image is updated to obtain a panoramic image with a higher resolution and the same picture at both ends.

[0100] Optionally, take a 4-way split image as an example: in the 4th split image, one part is on the left side of the panoramic image, and the other part is on the right side of the panoramic image. It just so happens that the leftmost and rightmost sides of the panoramic image can be connected, and the image data is also continuous in the encoder memory (the first pixel of a row is immediately followed by the last pixel of the previous row).

[0101] Optionally, the stride setting method can be used to encode the fourth stream, but there will be a line misalignment (the left side of the first line follows the right side of the second line), which is not obvious in the picture.

[0102] Optionally, if resource consumption is not considered, since the right side of the panoramic screen of the fourth video stream crosses to the left side of the panoramic screen, a row pixel offset may occur. Therefore, if a flat screen that crosses the border of the panoramic screen is generated, the left side of the panoramic screen can be directly spliced ​​to the right side before the panoramic image is diverted to ensure that the flat screen does not cross the spliced ​​panoramic screen. The size of the missing area can be determined according to the position of the edge diverted image in the panoramic image, and then according to the size of the missing area, an image area of ​​the same size can be intercepted on the other side of the panoramic image and spliced ​​to the edge of the panoramic image to form a larger panoramic image, such as changing the panoramic screen from 360 degrees to 390 degrees; and then the image stride value in the image parameter value of the modified panoramic image is reset (the image format and the length and width dimensions of the target image do not need to be corrected).

[0103] Reference Figure 6 , Figure 6 Schematic diagram of functional modules of the first embodiment of the video splitting device based on panoramic images of the present invention.

[0104] In this embodiment, the panoramic image-based video splitting device includes:

[0105] A diversion image size setting module 10 is used to set the diversion image size according to the effective resolution of the panoramic image;

[0106] A starting coordinate setting module 20 is configured to set the starting coordinates of each diversion image according to the size of the diversion image, the size of the preset overlapping area, and the preset coordinate position of the diversion image in the panoramic image;

[0107] The split image output module 30 is configured to input the split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image into a row stride encoder to output the split image.

[0108] Furthermore, the split image output module 30 specifically includes:

[0109] an image data reading unit, configured to run the row stride encoder according to the starting coordinates of the split image to read image data of the panoramic image;

[0110] a valid data determining unit, configured to determine valid data in the image data read by the row stride encoder according to the size of the split image and the row pixel values ​​of the panoramic image;

[0111] The shunt image obtaining unit is configured to output the valid data through the row stride encoder as shunt image data to obtain the shunt image.

[0112] Furthermore, the valid data determination unit specifically includes:

[0113] a coded data obtaining subunit, configured to sequentially read pixels of panoramic image rows through the row stride encoder to obtain coded data of the panoramic image;

[0114] An effective row pixel determination subunit, configured to determine effective row pixels of each row of the diversion image in a corresponding row of the panoramic image according to the starting coordinates of the diversion image and the size of the diversion image;

[0115] The valid data determination subunit is configured to determine valid data corresponding to the split image in the panoramic image coded data according to the valid row of pixels.

[0116] Furthermore, the panoramic image-based video splitting device further includes an effective resolution determination module, which specifically includes:

[0117] an effective area determination unit, configured to screen all pixels of the panoramic image according to a preset distortion threshold, and determine an area where the distortion amount of the pixels is within the distortion threshold as a effective area;

[0118] The effective resolution determining unit is configured to determine the effective resolution of the panoramic image according to the pixel values ​​contained in the effective area.

[0119] Furthermore, the offload image size setting module 10 specifically includes:

[0120] a row value determining unit, configured to determine the row value of the split image resolution according to the effective resolution of the panoramic image;

[0121] The column value determining unit is configured to calculate the proportion of the diverted image in the panoramic image according to a preset diverted angle, and determine the column value of the diverted image resolution.

[0122] Furthermore, the starting coordinate setting module 20 specifically includes:

[0123] a readable area determining unit, configured to determine a readable area of ​​the row stride encoder in the panoramic image according to the effective resolution of the panoramic image;

[0124] a starting coordinate setting unit, configured to set the starting coordinates of the diversion image in the panoramic image according to the readable area and the size of the diversion image;

[0125] The adjacent starting coordinate determining unit is configured to determine the starting coordinates of the adjacent diversion image of the diversion image according to the preset overlapping image size and the starting coordinates.

[0126] Furthermore, the panoramic image-based video splitting device further includes a panoramic image stitching module, which specifically includes:

[0127] a missing image size determining unit, configured to determine the missing image size of the edge diversion image according to the diversion image size, the preset overlapping image size, and the starting coordinates of the diversion image;

[0128] a supplementary image interception unit, configured to intercept a supplementary image having the same size as the missing image, starting from the first column on the other side of the panoramic image, according to the size of the missing image;

[0129] The image stitching unit is configured to align and stitch the supplementary image to the edge position of the panoramic image, and update the panoramic image so that the edge branch image is read completely.

[0130] Among them, each module in the above-mentioned video splitting device based on panoramic images corresponds to each step in the above-mentioned embodiment of the video splitting method based on panoramic images, and their functions and implementation processes are no longer described here one by one.

[0131] In addition, an embodiment of the present invention also provides a computer-readable storage medium.

[0132] The computer-readable storage medium of the present invention stores a video splitting program based on panoramic images, wherein when the video splitting program based on panoramic images is executed by a processor, the steps of the video splitting method based on panoramic images as described above are implemented.

[0133] The method implemented when the panoramic image-based video splitting program is executed can refer to the various embodiments of the panoramic image-based video splitting method of the present invention, and will not be described in detail here.

[0134] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0135] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0136] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0137] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0138] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video streaming method based on panoramic images, characterized in that: The steps of the method include: Setting the size of the split image according to the effective resolution of the panoramic image; wherein the effective resolution includes the effective row pixel values ​​and the effective column pixel values ​​of the panoramic image; Setting the starting coordinates of each diversion image according to the diversion image size, the preset overlapping area size, and the preset coordinate position of the diversion image in the panoramic image; The size of the split image, the row pixel value of the panoramic image, and the starting coordinates of each split image are input into a row stride encoder to output the split image.

2. The video splitting method based on panoramic images according to claim 1, characterized in that: Inputting the size of the split image, the row pixel value of the panoramic image, and the starting coordinates of each split image into a row stride encoder to output the split image, including: Running the row stride encoder according to the starting coordinates of the split image to read image data of the panoramic image; Determining valid data in the image data read by the row stride encoder according to the size of the split image and the row pixel values ​​of the panoramic image; The valid data is outputted through the row stride encoder as the shunt image data to obtain the shunt image.

3. The video splitting method based on panoramic images according to claim 2, characterized in that: The determining, according to the size of the shunted image and the row pixel value of the panoramic image, valid data in the image data read by the row stride encoder comprises: Reading the panoramic image row pixels in sequence through the row stride encoder to obtain panoramic image encoded data; Determining, according to the starting coordinates of the diversion image and the size of the diversion image, valid row pixels of each row of the diversion image in a corresponding row of the panoramic image; According to the valid row of pixels, valid data corresponding to the split image is determined in the panoramic image coded data.

4. The video splitting method based on panoramic images according to claim 1, characterized in that: Before setting the size of the split image according to the effective resolution of the panoramic image, the method further includes: Screening all pixels of the panoramic image according to a preset distortion threshold, and determining an area where the distortion amount of the pixels is within the distortion threshold as a valid area; The effective resolution of the panoramic image is determined according to the pixel values ​​contained in the effective area.

5. The video splitting method based on panoramic images according to claim 4, characterized in that: The step of setting the size of the split image according to the effective resolution of the panoramic image includes: Determining the row pixel value of the split image resolution according to the effective resolution of the panoramic image; According to a preset diversion angle, the proportion of the diversion image in the panoramic image is calculated, and the column pixel value of the diversion image resolution is determined.

6. The video splitting method based on panoramic images according to claim 5, characterized in that: The step of setting the starting coordinates of each diversion image according to the diversion image size, the preset overlapping area size, and the preset coordinate position of the diversion image in the panoramic image includes: Determining a readable area of ​​the row stride encoder in the panoramic image according to the effective resolution of the panoramic image; Setting the starting coordinates of the diversion image in the panoramic image according to the readable area and the size of the diversion image; The starting coordinates of adjacent diversion images of the diversion image are determined according to the preset overlapping area size and the starting coordinates.

7. The video splitting method based on panoramic images according to claim 1, characterized in that: Before inputting the size of the split image, the row pixel value of the panoramic image, and the starting coordinates of each split image into a row stride encoder to output the split image, the method further includes: Determining the missing image size of the edge diversion image according to the diversion image size, the preset overlapping area size, and the starting coordinates of the diversion image; According to the size of the missing image, starting from the first column on the other side of the panoramic image, intercepting a supplementary image with the same size as the missing image; The supplementary image is aligned and spliced ​​to the edge position of the panoramic image, and the panoramic image is updated so that the edge branch image is read completely.

8. A video splitting device based on panoramic images, characterized in that: The video splitting device based on panoramic images includes: A split image size setting module, configured to set the split image size according to the effective resolution of the panoramic image; wherein the effective resolution includes the effective row pixel values ​​and the effective column pixel values ​​of the panoramic image; a starting coordinate setting module, configured to set the starting coordinates of each diversion image according to the size of the diversion image and the preset coordinate position of the diversion image in the panoramic image; The split image output module is configured to input the split image size, the row pixel value of the panoramic image, and the starting coordinates of each split image into a row stride encoder to output the split image.

9. A video streaming device based on panoramic images, characterized in that: The panoramic image-based video diversion device includes a processor, a memory, and a panoramic image-based video diversion program stored on the memory and executable by the processor, wherein when the panoramic image-based video diversion program is executed by the processor, the steps of the panoramic image-based video diversion method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a video splitting program based on panoramic images, wherein when the video splitting program based on panoramic images is executed by a processor, the steps of the video splitting method based on panoramic images as described in any one of claims 1 to 7 are implemented.

Images