Video stream cutting method and device, computer equipment and readable storage medium

Abstract

The invention relates to a video stream cutting method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring multiple frames of to-be-clipped video images contained in a to-be-clipped target video stream and image clipping information corresponding to the multiple frames of to-be-clipped video images; inputting each to-be-clipped video image and the image clipping information into an OpenGLES framework which completes initialization processing in advance, and performing image clipping processing on each to-be-clipped video image based on the image clipping information through the OpenGLES framework to obtain a clipped video image corresponding to each to-be-clipped video image; and outputting a cropped target video stream according to each cropped video image. By the adoption of the method, image cutting can be completed through an OpenGLES framework, and multiple video images can be cut only by completing one-time initialization operation, so that the video stream cutting method provided by the invention can reduce system resources occupied in the video stream cutting process.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video stream cropping method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of image processing technology, a technology for cropping video streams has emerged. For example, for the captured camera fisheye image, the four corners with severe distortion need to be cropped and then displayed on the screen or processed again.

[0003] In traditional technology, in the Android development framework, you can use the Bitmap + Canvas + Paint classes to crop a single image. Therefore, when processing the video stream, you need to convert the video data image of the camera or player into RGB first, and then create a corresponding set of Bitmap + Canvas + Paint for each frame of the image for processing.

[0004] However, when the video stream is cropped through the above framework, if it is implemented through the Bitmap + Canvas + Paint method, since each frame of the image needs to create a corresponding set of Bitmap + Canvas + Paint for processing, it is very performance-consuming, and it will also cause memory jitter and system freezes due to frequent memory application and release. Therefore, in the current technical solutions for cropping video streams, the video stream cropping process occupies more system resources. Summary of the invention

[0005] Based on this, it is necessary to provide a video stream cropping method, apparatus, computer equipment, computer-readable storage medium and computer program product that can reduce the system resources occupied by video stream cropping in order to address the above technical problems.

[0006] In a first aspect, the present application provides a video stream cropping method, comprising:

[0007] Acquire multiple frames of video images to be cropped contained in the target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped;

[0008] Inputting each of the video images to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, performing image cropping processing on each of the video images to be cropped based on the image cropping information through the OpenGLES framework, and obtaining a cropped video image corresponding to each of the video images to be cropped;

[0009] Output a cropped target video stream according to each of the cropped video images.

[0010] In one of the embodiments, the image cropping information includes image cropping coordinate information and image window size information corresponding to the cropped video image; the image cropping processing is performed on each of the video images to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to each of the video images to be cropped, including: obtaining texture coordinate information matching the image cropping coordinate information through the OpenGLES framework, and constructing an image window matching the image window size information; obtaining a sub-video image matching the texture coordinate information from each of the video images to be cropped; and drawing each of the sub-video images in the image window to obtain a cropped video image corresponding to each of the video images to be cropped.

[0011] In one of the embodiments, obtaining texture coordinate information matching the image cropping coordinate information through the OpenGLES framework includes: obtaining an image size corresponding to the video image to be cropped through the OpenGLES framework; and normalizing the image cropping coordinate information based on the image size to obtain the texture coordinate information.

[0012] In one of the embodiments, drawing each of the sub-video images in the image window to obtain a cropped video image corresponding to each of the video images to be cropped includes: drawing each of the sub-video images in the image window, and storing each image drawn in the image window into an off-screen buffer corresponding to the image window; and reading each of the drawn images from the off-screen buffer as each of the cropped video images.

[0013] In one of the embodiments, the step of inputting each of the video images to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, performing image cropping processing on each of the video images to be cropped based on the image cropping information through the OpenGLES framework, and obtaining a cropped video image corresponding to each of the video images to be cropped, includes: inputting the image cropping information into an OpenGLES framework that has been initialized in advance, and obtaining a current video image to be cropped; the current video image to be cropped is any one of the video images to be cropped; inputting the current video image to be cropped into an OpenGLES framework that has been initialized in advance and has been input with the image cropping information LES framework, performing image cropping processing on the current video image to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to the current video image to be cropped; when the current video image to be cropped is not the last frame of the video image to be cropped contained in the target video stream, taking the next frame of the video image to be cropped of the current video image to be cropped as the new current video image to be cropped, and returning to execute the step of inputting the current video image to be cropped into the OpenGLES framework that has been initialized and input with the image cropping information in advance, until the current video image to be cropped is the last frame of the video image to be cropped contained in the target video stream.

[0014] In one of the embodiments, before inputting each of the video images to be cropped and the image cropping information into the OpenGLES framework that has been initialized in advance, it also includes: opening the connection between the display service and the EGL interface included in the OpenGLES framework, and initializing the EGL interface; the display service is used to display the cropped target video stream; rendering configuration processing is performed on the initialized EGL interface to complete the initialization processing of the rendering environment of the OpenGLES framework; calling the GLES interface included in the OpenGLES framework to create a shader object and a program object to complete the initialization processing of the shader of the OpenGLES framework; after the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework are initialized, the initialization processing of the OpenGLES framework is completed.

[0015] In one of the embodiments, after outputting the cropped target video stream according to each of the cropped video images, it also includes: releasing the shader object and the program object to release the shader of the OpenGLES framework; releasing the rendering configuration of the EGL interface, and closing the connection between the display service and the EGL interface included in the OpenGLES framework to release the rendering environment of the OpenGLES framework; after the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework are released, completing the release process of the OpenGLES framework.

[0016] In a second aspect, the present application further provides a video stream cropping device, comprising:

[0017] A cropping information acquisition module, used to acquire multiple frames of video images to be cropped contained in a target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped;

[0018] A cropped image acquisition module, used for inputting each of the video images to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, and performing image cropping processing on each of the video images to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to each of the video images to be cropped;

[0019] The cropped video generation module is used to output a cropped target video stream according to each of the cropped video images.

[0020] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of the embodiments of the first aspect when executing the computer program.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0022] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0023] The above-mentioned video stream cropping method, device, computer equipment, computer-readable storage medium and computer program product obtain multiple frames of video images to be cropped contained in the target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped; input each video image to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, and perform image cropping processing on each video image to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to each video image to be cropped; and output a cropped target video stream according to each cropped video image. The present application obtains multiple frames of video images to be cropped and corresponding cropping information contained in the target video stream, and then can input the above-mentioned frames of video images to be cropped and the cropping information into the OpenGLES framework that has completed the initialization processing in advance to complete the image cropping and thus generate the cropped target video stream. Compared with the prior art that each frame of the image needs to create a set of corresponding Bitmap + Canvas + Paint for processing, the image cropping completed by the present application through the OpenGLES framework only needs to complete one initialization operation, and multiple video images can be cropped. Therefore, the video stream cropping method provided by the present application can reduce the system resources occupied by the video stream cropping process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a schematic flow chart of a video stream cropping method in one embodiment;

[0026] Figure 2 is a schematic diagram of a process of obtaining each cropped video image in an embodiment;

[0027] Figure 3 is a schematic diagram of a process of obtaining each cropped video image in another embodiment;

[0028] Figure 4 A schematic diagram of a process for initializing an OpenGLES framework in one embodiment;

[0029] Figure 5 It is a flowchart of a method for cutting video streams based on OpenGLES in one embodiment;

[0030] Figure 6is a structural block diagram of a video stream cutting device in one embodiment;

[0031] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] In one embodiment, Figure 1 As shown, a video stream cropping method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0034] Step S101, obtaining a plurality of frames of video images to be cropped contained in a target video stream to be cropped, and image cropping information corresponding to the plurality of frames of video images to be cropped.

[0035] The target video stream refers to the video stream that needs to be cropped. The video stream can be composed of multiple frames of video images, and the video frame images that constitute the target video stream are the video images to be cropped. The image cropping information is information used to indicate how to crop the video images of the target video stream. For example, the information can be information used to indicate the cropping area, and the information can be input by the user as needed.

[0036] Specifically, when cropping the target video stream, the terminal may first perform frame processing on the target video stream to obtain multiple frames of video images to be cropped, and may also obtain image cropping information input by the user for setting the cropping method of the target video stream.

[0037] Step S102, inputting each video image to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, performing image cropping processing on each video image to be cropped based on the image cropping information through the OpenGLES framework, and obtaining a cropped video image corresponding to each video image to be cropped.

[0038] The OpenGLES framework (Open Graphics Library for Embedded Systems) refers to a standard image rendering interface framework for embedded platforms that can run on GPUs. OpenGLES is a 3D graphics API designed for embedded devices, mainly used in devices such as mobile phones, PDAs, and game consoles. It reduces many unnecessary methods and data types in OpenGL, making it more efficient to run on embedded devices. The cropped video image refers to the video image obtained after image cropping.

[0039] Specifically, after the terminal obtains multiple frames of video images to be cropped and the corresponding image cropping information, it can input each frame of video image to be cropped and the image cropping information into the OpenGLES framework after the initialization processing is completed. Through this framework, each frame of video image to be cropped can be cropped based on the image cropping information, thereby obtaining the cropped video image corresponding to each frame of video image to be cropped.

[0040] Step S103: output a cropped target video stream according to each cropped video image.

[0041] After the cropping of each frame of video image is completed, each frame of the cropped video image can be output, thereby outputting the cropped target video stream.

[0042] In the above-mentioned video stream cropping method, by obtaining multiple frames of video images to be cropped contained in the target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped; each video image to be cropped and the image cropping information are input into an OpenGLES framework that has completed initialization processing in advance, and each video image to be cropped is subjected to image cropping processing based on the image cropping information through the OpenGLES framework to obtain cropped video images corresponding to each video image to be cropped; and the cropped target video stream is output according to each cropped video image. The present application obtains multiple frames of video images to be cropped contained in the target video stream, and corresponding cropping information, and then the above-mentioned frames of video images to be cropped and the cropping information can be input into an OpenGLES framework that has completed initialization processing in advance to complete image cropping and generate a cropped target video stream. Compared with the prior art, each frame of the image needs to create a set of corresponding Bitmap + Canvas + Paint for processing. The image cropping completed by the present application through the OpenGLES framework only needs to complete one initialization operation to crop multiple video images. Therefore, the video stream cropping method provided by the present application can reduce the system resources occupied by the video stream cropping process.

[0043] In one embodiment, the image cropping information includes image cropping coordinate information and image window size information corresponding to the cropped video image; Figure 2 As shown, step S102 may further include:

[0044] Step S201, obtaining texture coordinate information matching the image clipping coordinate information through the OpenGLES framework, and constructing an image window matching the image window size information.

[0045] In this embodiment, the image cropping information may consist of two parts, namely, image cropping coordinate information and image window size information, wherein the image cropping coordinate information is used to characterize the image cropping area, and the coordinate information may refer to the vertex information of the image cropping area. For example, for a video image to be cropped with a size of 1280x800, with the upper left corner as the origin of the coordinate system, the image cropping coordinate information may be upper left [0, 40], lower left [0, 760], lower right [1280,760], upper right [1280, 40], then the image cropping area is a rectangular area composed of the four coordinates of upper left [0, 40], lower left [0, 760], lower right [1280, 760], and upper right [1280, 40].

[0046] The image window size information is used to represent the size information of the image window corresponding to the cropped video image. In this embodiment, the cropped video image can be displayed through an image window, and the image window size information is the size information of the image window. The texture coordinate information refers to the coordinate information under the texture coordinate system, which refers to the coordinate system used by the OpenGLES framework. The texture coordinate information can be converted based on the image cropping coordinate information.

[0047] Specifically, after the terminal inputs the image cropping information into the OpenGLES framework, the OpenGLES framework can convert the image cropping coordinate information contained in the image cropping information into texture coordinate information corresponding to the texture coordinate system, and can also construct an image window of appropriate size according to the image window size information.

[0048] Step S202, obtaining a sub-video image matching the texture coordinate information from each video image to be cropped;

[0049] Step S203: draw each sub-video image in the image window to obtain a cropped video image corresponding to each video image to be cropped.

[0050] The sub-video image refers to the video image that matches the texture coordinate information in the video image to be cropped, and can refer to the video image retained after cropping. After the OpenGLES framework obtains the texture coordinate information, it can obtain the sub-video image that matches the texture coordinate information from each frame of the video image to be cropped, and draw each frame of the sub-video image in the image window, thereby generating a cropped video image corresponding to each frame of the video image to be cropped.

[0051] In this embodiment, the image cropping information can be composed of image cropping coordinate information and image window size information corresponding to the cropped video image. The terminal can convert the image cropping coordinate information into texture coordinate information through the OpenGLES framework, and then obtain a sub-video image from each frame of the video image to be cropped, and draw it in the image window corresponding to the image window size information to generate each frame of the cropped video image. In this way, the user can set the cropping area and the size of the cropped image as needed, thereby improving the intelligence of video image cropping.

[0052] Furthermore, step S201 may further include: obtaining an image size corresponding to the video image to be cropped through the OpenGLES framework; and normalizing the image cropping coordinate information based on the image size to obtain texture coordinate information.

[0053] Among them, the image size refers to the size of the video image to be cropped. In this embodiment, the texture coordinate system of OpenGLES takes the upper left corner as the origin, and only has x and y coordinates, and the range is [0.0, 1.0]. To fully represent a picture, its coordinates are {{0.0, 0.0}, {0.0, 1.0}, {1.0, 1.0}, {1.0,0.0}}. Therefore, when performing the coordinate system conversion, the image cropping coordinate information needs to be normalized, and the normalization can be implemented based on the image size corresponding to the video image to be cropped.

[0054] For example, the image size of the video image to be cropped is 1280x800, and the image cropping coordinate information can be upper left [0,40], lower left [0, 760], lower right [1280, 760], upper right [1280, 40]. Then, when performing normalization, the x coordinate needs to be divided by 1280, and the y coordinate needs to be divided by 800. The resulting texture coordinate information is upper left [0.0, 0.05], lower left [0.0, 0.95], lower right [1.0, 0.95], and upper right [1.0, 0.05].

[0055] In this embodiment, the OpenGLES framework can also obtain the image size corresponding to the video image to be cropped, and normalize the coordinate information based on the image size to obtain texture coordinate information. In this way, the efficiency of texture coordinate acquisition can be improved.

[0056] In addition, step S203 may further include: drawing each sub-video image in the image window, and storing each image drawn in the image window into an off-screen buffer corresponding to the image window; reading each image drawn from the off-screen buffer as each cropped video image.

[0057] The off-screen buffer is a graphics processing technology for storing image data in a temporary memory outside the display screen. This technology is widely used in graphics rendering and image processing to improve rendering efficiency and graphics quality. In this embodiment, after each sub-video image is drawn in the image window, each image drawn in the image window can be stored in the off-screen buffer, and then the above-mentioned drawn image is read from the off-screen buffer and used as the cropped video image to complete the video cropping.

[0058] In this embodiment, after the sub-video image is drawn in the image window, the drawn image can be stored in an off-screen buffer and then read out from the off-screen buffer as a cropped video image that completes image cropping. In this way, the image rendering efficiency and graphics quality can be improved, that is, the cropping efficiency and cropping quality of the video image can be improved.

[0059] In one embodiment, Figure 3 As shown, step S102 may further include:

[0060] Step S301, inputting image cropping information into an OpenGLES framework that has completed initialization processing in advance, and obtaining a current video image to be cropped; the current video image to be cropped is any one of the video images to be cropped.

[0061] The current video image to be cropped refers to any frame of the video image to be cropped contained in the target video stream. In this embodiment, the input of multiple frames of video images to be cropped can be input into the OpenGLES framework frame by frame in the order of video frames, and the current video image to be cropped is the video image to be cropped currently input into the OpenGLES framework. Specifically, the terminal can first input the image cropping information into the OpenGLES framework that has completed the initialization processing in advance, and then determine the video image to be cropped that currently needs to be input into the OpenGLES framework, that is, determine the current video image to be cropped.

[0062] Step S302, input the current video image to be cropped into an OpenGLES framework that has been initialized and input with image cropping information, and perform image cropping processing on the current video image to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to the current video image to be cropped.

[0063] After determining the current video image to be cropped, the current video image to be cropped can be input into an OpenGLES framework that has been initialized in advance and carries image cropping information. Thereafter, the OpenGLES framework can be used to perform image cropping processing on the current video image to be cropped using the image cropping information to obtain a cropped video image corresponding to the current video image to be cropped.

[0064] Step S303, when the current video image to be cropped is not the last frame of the video image to be cropped contained in the target video stream, the next frame of the video image to be cropped of the current video image to be cropped is used as the new current video image to be cropped, and the process returns to step S302 until the current video image to be cropped is the last frame of the video image to be cropped contained in the target video stream.

[0065] If the current video image to be cropped is not the last frame of video image to be cropped contained in the target video stream, the terminal can use the next frame of video image to be cropped of the current video image to be cropped as the new current video image to be cropped, and input it into the OpenGLES framework that has completed the initialization processing and input the image cropping information in advance to complete the video image cropping. In this way, each video image to be cropped can be input into the OpenGLES framework in the order of the image frames to complete the image cropping, until the current video image to be cropped is the last frame of video image to be cropped contained in the target video stream, that is, at this time, the cropping of all video images to be cropped is completed.

[0066] In this embodiment, when cropping a video image, the video image to be cropped can be input into the OpenGLES framework in the order of video frames, so that the cropped video image is also output in the order of video frames. In this way, the accuracy of the order of each video image in the cropped target video stream can be ensured.

[0067] In one embodiment, Figure 4 As shown, before step S102, the following may also be included:

[0068] Step S401, opening the connection between the display service and the EGL interface included in the OpenGLES framework, and initializing the EGL interface; the display service is used to display the cropped target video stream.

[0069] The display service refers to the display application service used to display the cropped target video stream. The EGL interface, namely the Embedded-System Graphics Library interface, is an interface used to manage graphics contexts and surfaces, and is usually used with OpenGLES. The main function of EGL is to provide an underlying environment for OpenGLES, making the creation and management of graphics more efficient and flexible.

[0070] In this embodiment, when initializing the OpenGLES framework, it is first necessary to open the connection between the display service and the EGL interface included in the OpenGLES framework, and then the EGL interface can be initialized.

[0071] Step S402, performing rendering configuration processing on the initialized EGL interface to complete the initialization processing of the rendering environment of the OpenGLES framework.

[0072] Afterwards, the rendering configuration processing can be performed on the initialized EGL interface, and the rendering configuration can include the following processes: query and select the available rendering surface type and configuration, create a rendering context, create a rendering surface, and bind the "operating system display service" and "rendering context" and "rendering surface". Through the following steps, the rendering configuration of the initialized EGL interface can be completed, thereby completing the initialization processing of the rendering environment of the OpenGLES framework.

[0073] Step S403, calling the GLES interface included in the OpenGLES framework to create a shader object and a program object to complete the initialization processing of the shader of the OpenGLES framework;

[0074] Step S404, after the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework have completed the initialization process, the initialization process of the OpenGLES framework is completed.

[0075] After completing the initialization process for the rendering environment, it is also necessary to complete the initialization process for the shader of the OpenGLES framework in order to complete the initialization process for the OpenGLES framework. Among them, the initialization process for the shader of the OpenGLES framework can be implemented by calling the GLES interface contained in the OpenGLES framework. By calling the GLES interface, a shader object and a program object can be created, wherein the shader object is divided into a vertex shader and a fragment shader. The vertex shader is used to specify the vertex coordinate information of the image and the drawing window. In layman's terms, it is to tell the GPU what shape of image to draw. The fragment shader is used to tell the GPU how to sample each fragment and what the final color value is (it can be sampled and processed as the final color value); these two shaders are programmable. The program object is generated by compiling and linking the vertex shader and the fragment shader, through which data can be transferred between the application and the GPU, and instructions can be issued to notify the GPU to draw images. In this embodiment, the shader object and the program object can be created by calling the GLES interface contained in the OpenGLES framework, thereby completing the initialization process for the shader of the OpenGLES framework.

[0076] After the rendering environment and shader initialization processing of the OpenGLES framework are completed, the initialization processing of the OpenGLES framework can be completed, thereby preparing for inputting each video image to be cropped into the OpenGLES framework that has completed the initialization processing in advance.

[0077] In this embodiment, before the video image to be cropped is input, the initialization processing for the OpenGLES framework can be completed in advance. The initialization processing process includes the initialization processing of the rendering environment and shader of the OpenGLES framework. In this way, the efficiency of the initialization processing of the OpenGLES framework can be further improved.

[0078] In addition, after step S103, it can also include: releasing shader objects and program objects to release the shader of the OpenGLES framework; releasing the rendering configuration of the EGL interface, and closing the connection between the display service and the EGL interface included in the OpenGLES framework to release the rendering environment of the OpenGLES framework; after the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework are released, completing the release processing of the OpenGLES framework.

[0079] After completing the video stream cropping, if you need to release the OpenGLES framework, you need to release the shader and rendering environment of the OpenGLES framework accordingly. The release of the shader can be the release of resources such as shader objects and program objects, for example, the release of the created texture ID, program object, shader object, etc.

[0080] The release of the OpenGLES environment is the opposite of the initialization process. First, the rendering configuration of the EGL interface is released. For example, it can include: releasing the binding between the "operating system display service", "rendering context" and "rendering surface", releasing the rendering surface EGLSurface and releasing the rendering context EGLContext, and then closing the connection between the display service and the EGL interface contained in the OpenGLES framework to complete the release of the rendering environment of the OpenGLES framework. Through the above process, the release of the OpenGLES framework can be completed.

[0081] In this embodiment, after the video stream is cropped, the OpenGLES framework can also be released. In this way, after the video stream is cropped, the occupation of system resources by the OpenGLES framework can be reduced, thereby further improving the utilization rate of system resources.

[0082] In one embodiment, a video stream cropping method based on OpenGLES is also provided, wherein the core of the cropping method is to specify the texture coordinates of the input image and then specify the ViewPort (view window size) of the output image; then, the image corresponding to the coordinate range is drawn into the window according to the specified texture coordinates, and the image outside the coordinate range will not be drawn, so as to achieve the purpose of cropping. Figure 5 As shown, the method may include the following steps:

[0083] 1. Initialize the OpenGLES environment:

[0084] In the OpenGLES framework, there are two sets of interfaces, EGL (embedded Graphic Interface) and GLES. EGL is an application program interface for obtaining and managing the operating system drawing surface. In layman's terms, EGL is a set of standard interfaces used to create a running environment. The EGL interface is consistent on different operating systems, but its internal details are implemented by each system manufacturer. GLES is used to compile / link shader code, pass data to the GPU, and call drawing-related interfaces to notify the GPU to draw.

[0085] When creating an OpenGLES environment, the EGL interface is mainly called, and the following steps are required:

[0086] (1) Open the connection between EGL and the operating system display service;

[0087] (2) Initialize EGL;

[0088] (3) Query and select available rendering surface types and configurations;

[0089] (4) Create a rendering context;

[0090] (5) Create a rendering surface

[0091] (6) Bind the "operating system display service" to the "rendering context" and the "rendering surface".

[0092] 2. Initialize the shader:

[0093] During initialization, the GLES interface is mainly called. Under this framework, shader objects and program objects need to be created. Shaders are divided into vertex shaders and fragment shaders. Vertex shaders are used to specify the vertex coordinate information of the image and drawing window. In layman's terms, they tell the GPU what shape of image to draw. Fragment shaders are used to tell the GPU how to sample each fragment and what the final color value is (it can be sampled and processed before being used as the final color value). These two shaders are programmable. Program objects are generated by compiling and linking vertex shaders and fragment shaders. Through them, data can be transferred between the application and the GPU, and instructions can be issued to notify the GPU to draw images.

[0094] The function corresponding to the shader program object created in this embodiment is to redraw the image according to the input vertex coordinates (determine the shape of the image after drawing), texture coordinates (determine the sampling shape of the input image) and texture to generate a new image.

[0095] 3. Input image:

[0096] In OpenGLES, the memory of the image is allocated on the GPU, and the application has an index (hereinafter referred to as the texture ID) to indicate the address of this GPU memory. First, you need to create a texture ID, and when you need to transfer image data, bind texture textures[0] to the 2D texture.

[0097] 4. Crop the image:

[0098] The core idea of ​​cropping an image in this embodiment is to draw the cropped area onto the window by specifying texture coordinates and the size of the output window, thereby achieving the purpose of cropping.

[0099] Take a 1280x800 fisheye image as an example. Now we need to crop it to 1280x720 in the center. Then we need to crop 40 pixels on the upper and lower sides. The image takes the upper left corner as the origin of the coordinate system, and crops the image within the upper left [0, 40], lower left [0, 760], lower right [1280, 760], and upper right [1280, 40].

[0100] The texture coordinate system of OpenGLES takes the upper left corner as the origin, and has only x and y coordinates, both in the range of [0.0,1.0]. To fully represent an image, its coordinates are {{0.0, 0.0}, {0.0, 1.0}, {1.0, 1.0}, {1.0,0.0}}; so when you need to intercept the images in [0, 40], [0, 760], [1280, 760], [1280, 40] in the above case and quantize their coordinates into texture coordinates, the x coordinate needs to be divided by 1280 and the y coordinate needs to be divided by 800, and the obtained coordinates are: {{0.0, 0.05}, {0.0, 0.95}, {1.0, 0.95}, {1.0,0.05}}, and the width and height of the output image window need to be set to 1280x720. After calling the drawing code, you need to call the eglSwapBuffers function to "swap" the buffer.

[0101] 5. Export image or display image:

[0102] After drawing and before swapping buffers, you can call the glReadPixels function to read the drawn image from the GPU. If you need to display the image directly, you can specify the screen-related window structure in the EGLSurface created when initializing the OpenGLES environment described above. Then, after drawing is completed, call the eglSwapBuffers function, and the drawn data will be displayed on the screen by the operating system.

[0103] 6. Release the shader:

[0104] When releasing the shader, you need to release the texture ID, program object, shader object and other resources created above.

[0105] 7. Release the OpenGLES environment:

[0106] The order of release is opposite to that of initialization, and the order is as follows:

[0107] (1) Unbind the operating system display service from the rendering context and the rendering surface;

[0108] (2) Release the rendering surface EGLSurface;

[0109] (3) Release the rendering context EGLContext;

[0110] (4) Close the connection EGLDisplay between EGL and the operating system.

[0111] The video stream cropping method based on OpenGLES provided in this embodiment can realize image cropping based on the standard embedded operating system open graphics library standard interface OpenGLES, calculate the texture coordinates of the input image according to the size to be cropped, and specify the output window size; then read the image within the coordinate range from the input image according to the texture coordinates through a programmable shader and draw it to the output window; then read the processed image from the off-screen buffer corresponding to the output window or directly display it on the screen. This implementation method does not require the transplantation of a large amount of third-party code, has low development difficulty, and a small amount of implemented code. It can be transplanted in most mobile operating systems; because it runs on the GPU, it has high performance and is much more efficient than other image processing frameworks. It does not take up too many system resources while realizing the video data cropping function; developers can also implement cropping in the background and then read data according to their needs or directly display the data on the system window after cropping, which is flexible to use.

[0112] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a video stream cropping device for implementing the video stream cropping method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video stream cropping device embodiments provided below can refer to the limitations of the video stream cropping method above, and will not be repeated here.

[0114] In one embodiment, Figure 6As shown, a video stream cropping device is provided, including: a cropping information acquisition module 601, a cropped image acquisition module 602 and a cropped video generation module 603, wherein:

[0115] The cropping information acquisition module 601 is used to acquire multiple frames of video images to be cropped contained in the target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped;

[0116] The cropped image acquisition module 602 is used to input each video image to be cropped and the image cropping information into the OpenGLES framework that has completed the initialization process in advance, and perform image cropping processing on each video image to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to each video image to be cropped;

[0117] The cropped video generation module 603 is used to output a cropped target video stream according to each cropped video image.

[0118] In one embodiment, the image cropping information includes image cropping coordinate information and image window size information corresponding to the cropped video image; the cropped image acquisition module 602 is further used to obtain texture coordinate information matching the image cropping coordinate information through the OpenGLES framework, and construct an image window matching the image window size information; obtain a sub-video image matching the texture coordinate information from each video image to be cropped; draw each sub-video image in the image window to obtain a cropped video image corresponding to each video image to be cropped.

[0119] In one embodiment, the cropped image acquisition module 602 is further used to obtain the image size corresponding to the to-be-cropped video image through the OpenGLES framework; based on the image size, the image cropping coordinate information is normalized to obtain texture coordinate information.

[0120] In one embodiment, the cropped image acquisition module 602 is further used to draw each sub-video image in the image window, and store each image drawn in the image window into an off-screen buffer corresponding to the image window; and read each image drawn from the off-screen buffer as each cropped video image.

[0121] In one embodiment, the cropped image acquisition module 602 is further used to input the image cropping information into an OpenGLES framework that has been initialized in advance, and obtain the current video image to be cropped; the current video image to be cropped is any one of the video images to be cropped; the current video image to be cropped is input into an OpenGLES framework that has been initialized in advance and input with the image cropping information, and the OpenGLES framework performs image cropping processing on the current video image to be cropped based on the image cropping information to obtain a cropped video image corresponding to the current video image to be cropped; when the current video image to be cropped is not the last frame of the video image to be cropped contained in the target video stream, the next frame of the video image to be cropped of the current video image to be cropped is used as the new current video image to be cropped, and the step of inputting the current video image to be cropped into the OpenGLES framework that has been initialized in advance and input with the image cropping information is returned to execute, until the current video image to be cropped is the last frame of the video image to be cropped contained in the target video stream.

[0122] In one embodiment, the video stream cropping device also includes: an OpenGLES framework initialization module, which is used to open the connection between the display service and the EGL interface included in the OpenGLES framework, and initialize the EGL interface; the display service is used to display the cropped target video stream; the initialized EGL interface is subjected to rendering configuration processing to complete the initialization processing of the rendering environment of the OpenGLES framework; the GLES interface included in the OpenGLES framework is called to create a shader object and a program object to complete the initialization processing of the shader of the OpenGLES framework; after the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework are initialized, the initialization processing of the OpenGLES framework is completed.

[0123] In one embodiment, the video stream clipping device also includes: an OpenGLES framework release module, which is used to release shader objects and program objects to release the shader of the OpenGLES framework; release the rendering configuration of the EGL interface, and close the connection between the display service and the EGL interface included in the OpenGLES framework to release the rendering environment of the OpenGLES framework; after the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework are released, complete the release process of the OpenGLES framework.

[0124] Each module in the above video stream cutting device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0125] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a video stream cutting method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0126] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video stream cropping method, characterized in that: The method comprises: Acquire multiple frames of video images to be cropped contained in the target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped; Inputting each of the video images to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, performing image cropping processing on each of the video images to be cropped based on the image cropping information through the OpenGLES framework, and obtaining a cropped video image corresponding to each of the video images to be cropped; Output a cropped target video stream according to each of the cropped video images.

2. The method according to claim 1, characterized in that The image cropping information includes image cropping coordinate information and image window size information corresponding to the cropped video image; The performing image cropping processing on each of the to-be-cropped video images based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to each of the to-be-cropped video images includes: Acquire texture coordinate information matching the image clipping coordinate information through the OpenGLES framework, and construct an image window matching the image window size information; Acquire a sub-video image matching the texture coordinate information from each of the video images to be cropped; Each of the sub-video images is drawn in the image window to obtain a cropped video image corresponding to each of the video images to be cropped.

3. The method according to claim 2, characterized in that The obtaining, through the OpenGLES framework, texture coordinate information matching the image clipping coordinate information comprises: Acquire the image size corresponding to the video image to be cropped through the OpenGLES framework; Based on the image size, the image clipping coordinate information is normalized to obtain the texture coordinate information.

4. The method according to claim 2, characterized in that: Drawing each of the sub-video images in the image window to obtain a cropped video image corresponding to each of the video images to be cropped includes: Drawing each of the sub-video images in the image window, and storing each image drawn in the image window into an off-screen buffer corresponding to the image window; The drawn images are read from the off-screen buffer as the cropped video images.

5. The method according to any one of claims 1 to 4, characterized in that: The step of inputting each of the video images to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, performing image cropping processing on each of the video images to be cropped based on the image cropping information through the OpenGLES framework, and obtaining a cropped video image corresponding to each of the video images to be cropped, comprises: Input the image cropping information into an OpenGLES framework that has completed initialization processing in advance, and obtain a current video image to be cropped; the current video image to be cropped is any one of the video images to be cropped; Inputting the current video image to be cropped into an OpenGLES framework that has been pre-initialized and input with the image cropping information, performing image cropping processing on the current video image to be cropped based on the image cropping information through the OpenGLES framework, and obtaining a cropped video image corresponding to the current video image to be cropped; In the case that the current video image to be cropped is not the last frame of the video image to be cropped contained in the target video stream, the next frame of the video image to be cropped of the current video image to be cropped is used as the new current video image to be cropped, and the step of inputting the current video image to be cropped into the OpenGLES framework that has been initialized and has the image cropping information is returned to execution until the current video image to be cropped is the last frame of the video image to be cropped contained in the target video stream.

6. The method according to claim 1, characterized in that Before inputting each of the to-be-cropped video images and the image cropping information into the OpenGLES framework that has completed the initialization process in advance, the method further includes: Opening a connection between a display service and an EGL interface included in the OpenGLES framework, and initializing the EGL interface; the display service is used to display the cropped target video stream; Performing rendering configuration processing on the initialized EGL interface to complete the initialization processing of the rendering environment of the OpenGLES framework; Calling the GLES interface included in the OpenGLES framework to create a shader object and a program object to complete the initialization processing of the shader of the OpenGLES framework; After the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework complete the initialization processing, the initialization processing of the OpenGLES framework is completed.

7. The method according to claim 6, characterized in that After outputting the cropped target video stream according to each of the cropped video images, the method further includes: Release the shader object and the program object to release the shader of the OpenGLES framework; Release the rendering configuration of the EGL interface, and close the connection between the display service and the EGL interface included in the OpenGLES framework to release the rendering environment of the OpenGLES framework; After the rendering environment of the OpenGLES framework and the shader of the OpenGLES framework are released, the release process of the OpenGLES framework is completed.

8. A video stream cutting device, characterized in that: The device comprises: A cropping information acquisition module, used to acquire multiple frames of video images to be cropped contained in a target video stream to be cropped, and image cropping information corresponding to the multiple frames of video images to be cropped; A cropped image acquisition module, used for inputting each of the video images to be cropped and the image cropping information into an OpenGLES framework that has been initialized in advance, and performing image cropping processing on each of the video images to be cropped based on the image cropping information through the OpenGLES framework to obtain a cropped video image corresponding to each of the video images to be cropped; The cropped video generation module is used to output a cropped target video stream according to each of the cropped video images.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images