Video image processing method and device, computer equipment and storage medium

Abstract

The invention provides a video image processing method and device, computer equipment and a storage medium. The method comprises the steps of displaying a target video image in response to a video image detection request; creating a drawing canvas, and determining an interested drawing area matched with the drawing canvas on the basis of historical coordinate data under the condition that the vertex coordinate data set contains the historical coordinate data; and based on the interested drawing area, performing drawing processing on the target video image in the drawing canvas to obtain a video image processing result. According to the method provided by the invention, rich and diversified regions of interest can be provided, the flexibility and diversity of video image processing are improved, and the performance of video image processing is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video image processing method, apparatus, computer equipment and storage medium. Background Art

[0002] In more and more application fields, after acquiring a real-time video stream, it is necessary to detect a region of interest in the real-time video stream and display the region corresponding to the region of interest in the real-time video stream to help users quickly focus on more useful areas in the real-time video stream.

[0003] In the related art, the canvas element is mainly used to draw the area of ​​interest of the real-time video stream, that is, the canvas size of the canvas element is first pre-set, then the coordinate information of the area of ​​interest of the real-time video stream to be drawn is determined, and the area of ​​interest of the real-time video stream is drawn in the canvas to obtain the area of ​​interest of the real-time video stream.

[0004] This drawing method makes the drawn area of ​​interest of the real-time video stream depend on the preset canvas size and the coordinate information of the area of ​​interest in the determined real-time video stream. This results in the drawn area of ​​interest of the real-time video stream being relatively simple and lacking in flexibility, which easily affects the detection / display effect of the area of ​​interest of the real-time video stream. Summary of the invention

[0005] The present application provides a video image processing method, apparatus, computer equipment and storage medium to solve the problem that the existing video image processing method has a single detection area and lacks flexibility, thereby affecting the detection effect of the video image corresponding to the real-time video stream.

[0006] In a first aspect, the present application provides a video image processing method, comprising:

[0007] In response to a video image detection request, displaying a target video image; wherein the video image detection request is used to request detection of an area of ​​interest of the target video image; the target video image indicates a frame image of a real-time video stream;

[0008] A drawing canvas is created, and when it is determined that the vertex coordinate data set includes historical coordinate data, a drawing region of interest matching the drawing canvas is determined based on the historical coordinate data; wherein the vertex coordinate data set is used to store the coordinates of the vertices included in the drawing region of interest; and the historical coordinate data indicates the coordinates of the drawing region of interest determined in response to the video image detection request last time;

[0009] Based on the drawing area of ​​interest, the target video image is drawn in the drawing canvas to obtain a video image processing result; wherein the video image processing result indicates an area in the target video image corresponding to the drawing area of ​​interest.

[0010] In one example, in response to the video image detection request, displaying the target video image includes:

[0011] In response to the video image detection request, determining a target container; wherein the target container is a container for displaying the target video image;

[0012] determining an image zoom ratio based on the container size information of the target container and the image size information of the frame image of the real-time video stream;

[0013] Based on the image scaling ratio, the frame image of the real-time video stream is resized to obtain the target video image, and the target video image is displayed in the target container.

[0014] In one example, the historical coordinate data indicates that the historical coordinate data corresponds to the target video image detected in the last response to the video image detection request; and determining the drawing area of ​​interest matching the drawing canvas based on the historical coordinate data includes:

[0015] Determine first coordinate data of the historical coordinate data based on the image zoom ratio; wherein the first coordinate data indicates coordinate data determined with the origin of the coordinate system corresponding to the target container as the vertex;

[0016] Based on the first coordinate data, determine a coordinate offset; wherein the coordinate offset indicates a minimum value of the first coordinate data on each coordinate axis in a coordinate system corresponding to the target container;

[0017] Based on the first coordinate data and the coordinate offset, second coordinate data are determined, and based on the second coordinate data, the drawing area of ​​interest is determined; wherein the second coordinate data indicates the coordinate data of the coordinates of the drawing area of ​​interest determined last in response to the video image detection request under the target container.

[0018] In one example, the method further includes:

[0019] When it is determined that the vertex coordinate data set does not contain historical coordinate data, an initial drawing area is created; wherein the initial drawing area indicates a central area of ​​the target video image;

[0020] The interesting drawing area is determined based on the initial drawing area.

[0021] In one example, determining the interested drawing area based on the initial drawing area includes:

[0022] In response to a mouse trigger operation, determining third coordinate data corresponding to the mouse trigger operation; wherein the third coordinate data indicates the coordinates of the vertex determined by the mouse trigger operation;

[0023] The interesting drawing region is determined based on the third coordinate data.

[0024] In one example, the method further includes:

[0025] In response to a video stream acquisition request, acquiring a new real-time video stream; wherein the video stream acquisition request is used to request to reacquire the real-time video stream;

[0026] The target video image is switched to display the frame image of the new real-time video stream, and the frame image of the new real-time video stream is processed to obtain a new video image processing result.

[0027] In one example, the method further includes:

[0028] Before displaying the frame image of the new real-time video stream, determining whether the resolution information of the new real-time video stream has changed;

[0029] When it is determined that the resolution information has changed, the frame image of the new real-time video stream is resized, and the frame image of the new real-time video stream is displayed after the resize.

[0030] In a second aspect, the present application provides a video image processing device, comprising:

[0031] A display unit, configured to display a target video image in response to a video image detection request; wherein the video image detection request is used to request detection of an area of ​​interest of the target video image; and the target video image indicates a frame image of a real-time video stream;

[0032] A determination unit, configured to create a drawing canvas, and, if it is determined that the vertex coordinate data set includes historical coordinate data, determine a drawing region of interest that matches the drawing canvas based on the historical coordinate data; wherein the vertex coordinate data set is used to store the coordinates of the vertices included in the drawing region of interest; and the historical coordinate data indicates the coordinates of the drawing region of interest determined in response to the video image detection request last time;

[0033] A processing unit is used to perform drawing processing on the target video image in the drawing canvas based on the drawing area of ​​interest to obtain a video image processing result; wherein the video image processing result indicates an area in the target video image corresponding to the drawing area of ​​interest.

[0034] In one example, the display unit is used to:

[0035] In response to the video image detection request, determining a target container; wherein the target container is a container for displaying the target video image;

[0036] determining an image zoom ratio based on the container size information of the target container and the image size information of the frame image of the real-time video stream;

[0037] Based on the image scaling ratio, the frame image of the real-time video stream is resized to obtain the target video image, and the target video image is displayed in the target container.

[0038] In one example, the historical coordinate data indicates that the historical coordinate data corresponds to the target video image detected in the last response to the video image detection request; and the determining unit is used to:

[0039] Determine first coordinate data of the historical coordinate data based on the image zoom ratio; wherein the first coordinate data indicates coordinate data determined with the origin of the coordinate system corresponding to the target container as the vertex;

[0040] Based on the first coordinate data, determine a coordinate offset; wherein the coordinate offset indicates a minimum value of the first coordinate data on each coordinate axis in a coordinate system corresponding to the target container;

[0041] Based on the first coordinate data and the coordinate offset, second coordinate data are determined, and based on the second coordinate data, the drawing area of ​​interest is determined; wherein the second coordinate data indicates the coordinate data of the coordinates of the drawing area of ​​interest determined last in response to the video image detection request under the target container.

[0042] In one example, the device is also used for:

[0043] When it is determined that the vertex coordinate data set does not contain historical coordinate data, an initial drawing area is created; wherein the initial drawing area indicates a central area of ​​the target video image;

[0044] The interesting drawing area is determined based on the initial drawing area.

[0045] In one example, the device is also used for:

[0046] In response to a mouse trigger operation, determining third coordinate data corresponding to the mouse trigger operation; wherein the third coordinate data indicates the coordinates of the vertex determined by the mouse trigger operation;

[0047] The interesting drawing region is determined based on the third coordinate data.

[0048] In one example, the device further includes a video acquisition unit, configured to:

[0049] In response to a video stream acquisition request, acquiring a new real-time video stream; wherein the video stream acquisition request is used to request to reacquire the real-time video stream;

[0050] The target video image is switched to display the frame image of the new real-time video stream, and the frame image of the new real-time video stream is processed to obtain a new video image processing result.

[0051] In one example, the device further includes a resolution detection unit, configured to:

[0052] Before displaying the frame image of the new real-time video stream, determining whether the resolution information of the new real-time video stream has changed;

[0053] When it is determined that the resolution information has changed, the frame image of the new real-time video stream is resized, and the frame image of the new real-time video stream is displayed after the resize.

[0054] In a third aspect, the present application provides a computer device, comprising: a processor, and a memory communicatively connected to the processor;

[0055] The memory stores computer-executable instructions;

[0056] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0058] In a fifth aspect, the present application provides a computer program product, comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of a computer device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program so that the computer device executes the method described in the first aspect.

[0059] The video image processing method, device, computer equipment and storage medium provided in the embodiment of the present application can display the frame image of the real-time video stream, that is, the target video image, in response to the video image detection request. At this time, the target video image can be used as the background to provide a drawing basis for the subsequent drawing of the region of interest. After that, a drawing canvas can be created, and it is determined whether there is historical coordinate data in the vertex coordinate data set corresponding to the video image detection request. In the case where it is determined that there is historical coordinate data, the drawing region of interest matching the drawing canvas is determined based on the historical coordinate data. In this way, the coordinates of the vertices of the drawing region of interest corresponding to various video image detection requests can be provided based on the vertex coordinate data set, so that the number of drawing regions of interest is more and richer, and the flexibility and diversity of video image processing can be improved. At the same time, this method can also map the coordinates of the vertices of the drawing region of interest determined last time that are pre-saved to the current drawing canvas, so that the drawing region of interest matching the current drawing canvas can be quickly determined, improving the speed and efficiency of video image processing. Finally, the target video image can be drawn and processed based on the drawing region of interest in the drawing canvas to obtain a video image processing result. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 A schematic diagram of a process for detecting an area of ​​interest of a frame image of a real-time video stream in a related technology provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of a process flow of a video image processing method provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of another video image processing method provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of an application flow of video image processing provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of an application scenario of video image processing provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of a video image processing device provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of another video image processing device provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0069] Fig. 9 The present invention is a block diagram of a computer device according to an exemplary embodiment.

[0070] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0071] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0072] Due to the characteristics of the human visual system, that is, people's attention is usually focused on a limited area in the frame image, while the sensitivity to other areas of the frame image is reduced. Therefore, in some application scenarios, such as video surveillance, telemedicine, and video calls, in order to help users focus more on more useful areas in the frame image, usually after acquiring the real-time video stream, the region of interest is detected for each frame image of the real-time video stream to display the area of ​​the real-time video stream corresponding to the region of interest, so that the user can quickly locate the region of interest.

[0073] Figure 1 The present invention provides a flow chart of detecting the region of interest of a frame image of a real-time video stream in a related technology according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0074] S101: Obtain a canvas generation request, where the canvas generation request includes a canvas size.

[0075] The canvas generation request is a request for instructing to generate a canvas element.

[0076] S102: Generate a canvas of a canvas size according to the canvas generation request.

[0077] S103: Establish a coordinate system with the upper left corner of the canvas as the origin, and obtain the object to be drawn and the drawing configuration information of the object to be drawn.

[0078] The object to be drawn indicates a frame image of a real-time video stream.

[0079] The drawing configuration information includes coordinates of the object to be drawn, and the coordinate values ​​in the coordinates include a first distance value of the object to be drawn moving downward with the origin as the center of the reference system, a second distance value of moving rightward, a third distance value of moving leftward with the maximum value of the x-axis as the center of the reference system, and at least one of a fourth distance value of moving upward with the maximum value of the y-axis as the center of the reference system.

[0080] S104: Determine a drawing starting point of the object to be drawn according to the coordinates of the drawing object included in the drawing configuration information.

[0081] S105 , drawing the object to be drawn at the drawing starting point in the canvas to obtain a drawing area.

[0082] This drawing method makes the drawn area of ​​interest of the real-time video stream more dependent on the preset canvas size. When the canvas size is fixed, the area of ​​interest of the real-time video stream will be relatively simple. In addition, if you want to change the area of ​​interest, you need to re-initiate the canvas generation request and then modify the size of the area of ​​interest, resulting in a lack of flexibility in drawing the area of ​​interest of the real-time video stream.

[0083] In addition, this drawing method cannot make the drawn area of ​​interest correspond to the real-time video stream at the current resolution when the real-time video stream changes, does not support the switching of real-time video streams with different resolutions, and has poor performance.

[0084] The video image processing method provided in this application is intended to solve the above technical problems in the prior art.

[0085] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0086] Figure 2 A schematic diagram of a video image processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0087] S201. In response to a video image detection request, display a target video image.

[0088] The video image detection request is used to request detection of an area of ​​interest of a target video image; the target video image indicates a frame image of a real-time video stream.

[0089] In one example, the region of interest of the target video image may indicate the target object contained in the target video image. For example, the target object may be: pedestrians, vehicles (e.g., cars), road signs, sky, fireworks, obstacles, etc. There is no limitation on the number and type of target objects here, which is subject to what can be achieved.

[0090] Different target objects require different regions of interest, and therefore correspond to different video image detection requests.

[0091] In one example, it is assumed that the real-time video stream is a video stream obtained in real time from a video surveillance device. Then, at this time, the real-time video stream can be used to request detection of an area of ​​interest of a frame image of the real-time video stream collected by the video surveillance, for example, pedestrians contained in the frame image can be detected.

[0092] At this time, in response to the video image detection request, the frame image of the real-time video stream collected by the video surveillance can be displayed in the target container.

[0093] S202: Create a drawing canvas, and when it is determined that the vertex coordinate data set includes historical coordinate data, determine a drawing region of interest that matches the drawing canvas based on the historical coordinate data.

[0094] The vertex coordinate data set is used to store the coordinates of the vertices included in the interested drawing area; the historical coordinate data indicates the coordinates of the interested drawing area determined in response to the video image detection request last time.

[0095] In an example, the created drawing canvas has the same size as the target container for displaying the target video image, so that the canvas can match the target container for displaying the target video image, so that the content drawn in the drawing canvas can be clearly and completely displayed in the target container, avoiding the problem of affecting the display effect due to the mismatch between the drawing canvas and the target container.

[0096] In one example, the canvas may be generated through an application programming interface of a package component. The application programming interface of the package component used is not limited here, and what can be implemented shall prevail.

[0097] In one example, the vertex coordinate data set is used to store the coordinates of the drawing area of ​​interest determined in the last response to the video image detection request corresponding to each video image detection request.

[0098] In one example, historical coordinate data corresponding to the current video image detection request may be determined from the vertex coordinate data set, and when it is determined that the corresponding historical coordinate data exists, the interested drawing area may be determined based on the corresponding historical coordinate data.

[0099] In one example, the historical coordinate data corresponds to the coordinate data in the frame image of the real-time video stream displayed in response to the video image detection request last time.

[0100] S203: Based on the drawing area of ​​interest, the target video image is drawn in the drawing canvas to obtain a video image processing result.

[0101] The video image processing result indicates an area in the target video image corresponding to the interested drawing area.

[0102] It can be seen from the above description that the video image processing method provided by the embodiment of the present application can display the frame image of the real-time video stream, that is, the target video image, in response to the video image detection request. At this time, the target video image can be used as a background to provide a drawing basis for the subsequent drawing of the region of interest. After that, a drawing canvas can be created, and it can be determined whether there is historical coordinate data in the vertex coordinate data set corresponding to the video image detection request. In the case where it is determined that there is historical coordinate data, the drawing region of interest matching the drawing canvas is determined based on the historical coordinate data. In this way, the coordinates of the vertices of the drawing region of interest corresponding to various video image detection requests can be provided based on the vertex coordinate data set, so that the number of drawing regions of interest is more and richer, and the flexibility and diversity of video image processing can be improved. At the same time, this method can also map the coordinates of the vertices of the drawing region of interest determined last time that are pre-saved to the current drawing canvas, so that the drawing region of interest matching the current drawing canvas can be quickly determined, improving the speed and efficiency of video image processing. Finally, the target video image can be drawn and processed based on the drawing region of interest in the drawing canvas to obtain a video image processing result.

[0103] Figure 3 A flowchart of another video image processing method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:

[0104] S301: In response to a video image detection request, determine a target container.

[0105] The target container is a container for displaying a target video image. For example, the target container may be an HTML (Hyper Text Markup Language) container.

[0106] S302: Determine an image zoom ratio based on the container size information of the target container and the image size information of the frame image of the real-time video stream.

[0107] Here, the container size information indicates the width and height of the target container, and the image size information indicates the width and height of the frame image of the real-time video stream.

[0108] At this time, a first ratio between the width of the target container and the width of the frame image of the real-time video stream can be calculated, and a second ratio between the height of the target container and the height of the frame image of the real-time video stream can be calculated. Then, the smaller value of the first ratio and the second ratio can be determined as the image scaling ratio.

[0109] S303: Based on the image scaling ratio, resize the frame image of the real-time video stream to obtain a target video image, and display the target video image in a target container.

[0110] In one example, the width and height of the frame image of the real-time video stream can be multiplied by the image scaling ratio to obtain the resized width and height. Then, the target video image can be determined based on the resized width and height.

[0111] At this time, the obtained target video image can be displayed in the target container and used as the background for video image processing.

[0112] This implementation can adjust the image size of the frame image of the real-time video stream according to the target container for displaying the target video image, so that the frame image of the real-time video stream matches the target container, thereby making full use of the display area of ​​the target container and completely and accurately displaying the frame image of the real-time video stream in the target container.

[0113] S304: Create a drawing canvas.

[0114] In one example, a canvas may be generated based on an application programming interface of a packaged component. For example, a drawing canvas may be created based on a fabric.js API.

[0115] In one example, after creating a drawing canvas, a vertex coordinate data set may be obtained, and it may be determined whether the vertex coordinate data set contains historical coordinate data corresponding to the video image detection request.

[0116] S305 . When it is determined that the vertex coordinate data set includes historical coordinate data, determine first coordinate data of the historical coordinate data based on the image zoom ratio.

[0117] The vertex coordinate data set is used to store the coordinates of the vertices included in the interested drawing area; the historical coordinate data indicates the coordinates of the interested drawing area determined in response to the video image detection request last time.

[0118] The first coordinate data indicates coordinate data determined by taking the origin of the coordinate system corresponding to the target container as the vertex.

[0119] The historical coordinate data indicates a target video image that corresponds to the target video image that was requested to be detected in response to the video image detection request last time.

[0120] S306: Determine a coordinate offset based on the first coordinate data.

[0121] The coordinate offset indicates the minimum value of the first coordinate data on each coordinate axis in the coordinate system corresponding to the target container. At this time, the number of coordinate offsets is associated with the number of coordinate axes included in the coordinate system corresponding to the target container.

[0122] S307 : Determine second coordinate data based on the first coordinate data and the coordinate offset, and determine a drawing region of interest based on the second coordinate data.

[0123] The second coordinate data indicates coordinate data of the coordinates of the drawing area of ​​interest determined in response to the video image detection request last time under the target container.

[0124] In one example, a Polygon instance may be established based on the first coordinate data and the coordinate offset to determine the second coordinate data and the drawing area of ​​interest.

[0125] In the above implementation, when it is determined that the vertex coordinate data set contains historical coordinate data, the historical coordinate data can be converted to second coordinate data matching the current target container through the image scaling ratio and the coordinate offset, and the vertex coordinates under the current target container are determined based on the second coordinate data to obtain the drawing area of ​​interest. Therefore, the last saved drawing area of ​​interest can be reused through the above coordinate conversion method, thereby simplifying the operation of determining the drawing area of ​​interest and improving the speed of determining the drawing area of ​​interest.

[0126] S308: When it is determined that the vertex coordinate data set does not contain historical coordinate data, an initial drawing area is created.

[0127] The initial drawing area indicates the central area of ​​the target video image.

[0128] At this time, the initial drawing area is an area obtained after presetting the drawing area type (eg, rectangle, circle, etc.) and drawing area attribute information (eg, center point coordinates, drawing area size, etc.).

[0129] For example, the initial drawing area may be a quadrilateral area with fixed width and height, and is located at the center of the drawing canvas by default.

[0130] S309: Determine a drawing area of ​​interest based on the initial drawing area.

[0131] In one example, the initial drawing area may be directly determined as the drawing area of ​​interest.

[0132] In another example, in response to a mouse trigger operation, third coordinate data corresponding to the mouse trigger operation may be determined, and the interested drawing area may be determined based on the third coordinate data.

[0133] The third coordinate data indicates the coordinates of the vertex determined by the mouse trigger operation.

[0134] In one example, the mouse-triggered operation is used to change the operation of the initial drawing area. For example, the mouse-triggered operation can be: moving the initial drawing area as a whole, dragging the vertices of the initial drawing area, changing the drawing area attribute information of the initial drawing area, changing the drawing area type of the initial drawing area, etc.

[0135] In one example, in addition to determining the third coordinate data in response to a mouse trigger operation, a trigger operation on the display device where the initial drawing area is located can also be detected to determine the third coordinate data corresponding to the trigger operation. The method for determining the third coordinate data is not limited here, so long as the third coordinate data can be determined arbitrarily.

[0136] In the above implementation, after determining the initial drawing area, the third coordinate data can be determined in response to the mouse trigger operation, so as to determine the drawing area of ​​interest based on the third coordinate data, so that the initial drawing area can be arbitrarily changed based on the mouse to determine the coordinates of the new vertices of the drawing area of ​​interest, which not only makes the method of determining the drawing area of ​​interest simpler and more accurate, but also further enhances the flexibility and diversity of the drawing area of ​​interest, and can meet the video image processing needs in more scenarios.

[0137] In an example, after performing video image processing on the displayed target video image according to the process described above and obtaining the video image processing result, the video image processing result can be viewed. If it is determined that the video image processing result does not meet the processing requirements, the process described in S310 below can be executed.

[0138] S310: In response to a video stream acquisition request, acquire a new real-time video stream.

[0139] The video stream acquisition request is used to request to re-acquire the real-time video stream.

[0140] In one example, a video acquisition button may be displayed in a display page corresponding to the target container, and after a trigger operation on the video acquisition button is detected, it is determined that a video stream acquisition request is detected. At this point, a new real-time video stream may be acquired in response to the video stream acquisition request.

[0141] In one example, when it is determined that the acquisition device for acquiring the real-time video stream has changed, the video acquisition button can be triggered to reacquire the real-time video stream. Alternatively, when it is necessary to switch the video image detection request, the video acquisition button can be triggered to reacquire the real-time video stream. The scenario for triggering the video acquisition button is not limited here.

[0142] In the above method, when the real-time video stream captured by the video stream capture device changes, or when the video image processing requirements for the real-time video stream that has been acquired change, a new real-time video stream can be acquired by responding to a video stream acquisition request, thereby conveniently and quickly updating the real-time video stream to better meet the video image processing requirements and improve the video image processing performance.

[0143] S311: Determine whether the resolution information of the new real-time video stream changes.

[0144] S312: When it is determined that the resolution information has changed, the frame image of the new real-time video stream is resized, and the frame image of the new real-time video stream is displayed after the resizing.

[0145] In one example, when the frame image of the new real-time video stream is resized, reference may be made to the process described in S301 to S303 above, which will not be described in detail here.

[0146] S313: Switch the target video image to display the frame image of the new real-time video stream, and process the frame image of the new real-time video stream to obtain a new video image processing result.

[0147] In the above implementation, before displaying the frame image of the new real-time video stream, it can be determined whether the resolution information of the frame image of the real-time video stream has changed, thereby ensuring that the frame image of the new real-time video stream is fully and clearly displayed in the current target container, which helps to ensure the accuracy of the new video image processing result after processing the frame image of the new real-time video stream.

[0148] Figure 4 The following is a schematic diagram of an application flow of video image processing provided by an embodiment of the present application. Figure 4 As shown, after detecting the video image detection request, the target video image can be displayed in response to the video image detection request. At this time, the target video image can be displayed in the target container. Then, a drawing canvas matching the target container can be created.

[0149] Next, you can first determine whether the vertex coordinate data set contains historical coordinate data. If it is determined that it is contained, you can determine the drawing area of ​​interest that matches the drawing canvas based on the historical coordinate data; if it is determined that it is not contained, you can first create an initial drawing area and detect the mouse trigger operation, so as to determine the drawing area of ​​interest in response to the mouse trigger operation.

[0150] Afterwards, the video acquisition request can be detected.

[0151] In the case where no video acquisition request is detected, the target video image may be drawn based on the interested drawing area to obtain a video image processing result.

[0152] After detecting the video acquisition request, a new real-time video stream is acquired in response to the video acquisition request. Then, it is detected whether the resolution information of the new real-time video stream has changed. If it is determined that the resolution information has changed, the frame image of the new real-time video stream can be resized by the image scaling ratio, and the frame image of the new real-time video stream can be displayed after the adjustment. After that, the frame image of the displayed new real-time video stream can be used as the target video image to perform video image processing to obtain a video image processing result.

[0153] When it is determined that the resolution information has not changed, the frame image of the new real-time video stream can be directly displayed, and the frame image of the new real-time video stream is used as the target video image to perform video image processing to obtain a video image processing result.

[0154] Figure 5 A schematic diagram of an application scenario of a video image processing method provided in an embodiment of the present application. The video image processing method provided in the present application can be applied to an AI (Artificial Intelligence) server.

[0155] like Figure 5 As shown, the server can send the video image processing task to the AI ​​server. At this time, the AI ​​server can add the received video image processing task to the database.

[0156] Then, the AI ​​server can forward the video image processing task to the gateway. At this time, after receiving the video image processing task, the gateway can request the video image that needs to be processed and analyzed from the streaming media server (or, video stream acquisition device).

[0157] At this time, the gateway can receive the video image to be processed sent by the streaming media server to obtain the video image, and process the video image to obtain a video processing result.

[0158] In one example, the gateway can send all video processing results to the AI ​​server and save the video processing results on the AI ​​server, so that subsequent staff can view them easily.

[0159] In another example, the gateway can also analyze the video processing results, and after detecting abnormal video processing results, report the abnormal video processing results to the AI ​​server. At this time, the AI ​​server can store the received abnormal video processing results in the database to facilitate subsequent staff to view the abnormal video processing results.

[0160] When detecting that the server has sent a termination message for the video image processing task to the AI ​​server, the AI ​​server can delete the corresponding video image processing task in the database. At this time, the gateway can send a task completion request to the streaming media server. After receiving the request, the streaming media server stops sending video images (i.e., the video images to be processed) to the gateway.

[0161] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0162] Figure 6 A schematic diagram of the structure of a video image processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the video image processing device 600 includes:

[0163] The display unit 601 is used to display a target video image in response to a video image detection request; wherein the video image detection request is used to request detection of an area of ​​interest of the target video image; the target video image indicates a frame image of a real-time video stream.

[0164] The determination unit 602 is used to create a drawing canvas, and when it is determined that the vertex coordinate data set includes historical coordinate data, determine a drawing area of ​​interest that matches the drawing canvas based on the historical coordinate data; wherein the vertex coordinate data set is used to store the coordinates of the vertices included in the drawing area of ​​interest; and the historical coordinate data indicates the coordinates of the drawing area of ​​interest determined in response to the video image detection request last time.

[0165] The processing unit 603 is used to perform drawing processing on the target video image in the drawing canvas based on the drawing area of ​​interest to obtain a video image processing result; wherein the video image processing result indicates an area in the target video image corresponding to the drawing area of ​​interest.

[0166] Figure 7A schematic diagram of the structure of another video image processing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the video image processing device 700 includes:

[0167] The display unit 701 is used to display a target video image in response to a video image detection request; wherein the video image detection request is used to request detection of an area of ​​interest of a target video image; and the target video image indicates a frame image of a real-time video stream.

[0168] In one example, the display unit 701 is used to determine a target container in response to a video image detection request; wherein the target container is a container for displaying a target video image; determine an image zoom ratio based on container size information of the target container and image size information of a frame image of a real-time video stream; based on the image zoom ratio, resize the frame image of the real-time video stream to obtain a target video image, and display the target video image in the target container.

[0169] The determination unit 702 is used to create a drawing canvas, and when it is determined that the vertex coordinate data set includes historical coordinate data, determine a drawing area of ​​interest that matches the drawing canvas based on the historical coordinate data; wherein the vertex coordinate data set is used to store the coordinates of the vertices included in the drawing area of ​​interest; and the historical coordinate data indicates the coordinates of the drawing area of ​​interest determined in response to the video image detection request last time.

[0170] In one example, the determination unit 702 is used to determine first coordinate data of historical coordinate data based on an image zoom ratio; wherein the first coordinate data indicates coordinate data determined with the origin of a coordinate system corresponding to a target container as a vertex; based on the first coordinate data, determine a coordinate offset; wherein the coordinate offset indicates the minimum value of the first coordinate data on each coordinate axis in the coordinate system corresponding to the target container; based on the first coordinate data and the coordinate offset, determine second coordinate data, and determine a drawing area of ​​interest based on the second coordinate data; wherein the second coordinate data indicates coordinate data of the coordinates of the drawing area of ​​interest determined in response to a video image detection request last time under the target container.

[0171] The processing unit 703 is used to perform drawing processing on the target video image in the drawing canvas based on the drawing area of ​​interest to obtain a video image processing result; wherein the video image processing result indicates an area in the target video image corresponding to the drawing area of ​​interest.

[0172] In one example, the device is also used to:

[0173] When it is determined that the vertex coordinate data set does not contain historical coordinate data, an initial drawing area is created; wherein the initial drawing area indicates a central area of ​​a target video image; and a drawing area of ​​interest is determined based on the initial drawing area.

[0174] In one example, the device is also used to:

[0175] In response to a mouse trigger operation, third coordinate data corresponding to the mouse trigger operation is determined; wherein the third coordinate data indicates the coordinates of the vertex determined by the mouse trigger operation; and a drawing area of ​​interest is determined based on the third coordinate data.

[0176] In one example, the device further includes:

[0177] The video acquisition unit 704 is used to acquire a new real-time video stream in response to a video stream acquisition request; wherein the video stream acquisition request is used to request to re-acquire the real-time video stream; the target video image is switched to display a frame image of the new real-time video stream, and the frame image of the new real-time video stream is processed to obtain a new video image processing result.

[0178] In one example, the device further includes:

[0179] The resolution detection unit 705 is used to determine whether the resolution information of the new real-time video stream has changed before displaying the frame image of the new real-time video stream; if it is determined that the resolution information has changed, the frame image of the new real-time video stream is resized and the frame image of the new real-time video stream is displayed after the adjustment.

[0180] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0181] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the computer device 800 includes: a memory 801 and a processor 802 .

[0182] Memory 801 is a memory used to store computer-executable instructions of processor 802.

[0183] The processor 802 is configured to execute the method provided in the above embodiment.

[0184] The computer device further includes a receiver 803 and a transmitter 804. The receiver 803 is used to receive instructions and data sent by an external device, and the transmitter 804 is used to send instructions and data to the external device.

[0185] Fig. 9It is a block diagram of a computer device according to an exemplary embodiment. The device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0186] Device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0187] The processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0188] The memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0189] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 900.

[0190] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0191] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0192] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0193] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display and keypad of the device 900, and the sensor assembly 914 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0194] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0195] In an exemplary embodiment, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0196] The present application also provides a computer-readable storage medium on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the steps of the video image processing method in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0197] The embodiment of the present application also provides a computer program product, which carries a computer program. The instructions included in the computer program can be used to execute the steps of the video image processing method in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0198] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0199] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video image processing method, characterized in that: include: In response to a video image detection request, displaying a target video image; wherein the video image detection request is used to request detection of an area of ​​interest of the target video image; the target video image indicates a frame image of a real-time video stream; A drawing canvas is created, and when it is determined that the vertex coordinate data set includes historical coordinate data, a drawing region of interest matching the drawing canvas is determined based on the historical coordinate data; wherein the vertex coordinate data set is used to store the coordinates of the vertices included in the drawing region of interest; and the historical coordinate data indicates the coordinates of the drawing region of interest determined in response to the video image detection request last time; Based on the drawing area of ​​interest, the target video image is drawn in the drawing canvas to obtain a video image processing result; wherein the video image processing result indicates an area in the target video image corresponding to the drawing area of ​​interest.

2. The method according to claim 1, characterized in that The step of displaying a target video image in response to a video image detection request includes: In response to the video image detection request, determining a target container; wherein the target container is a container for displaying the target video image; Determining an image zoom ratio based on container size information of the target container and image size information of the frame image of the real-time video stream; Based on the image scaling ratio, the frame image of the real-time video stream is resized to obtain the target video image, and the target video image is displayed in the target container.

3. The method according to claim 2, characterized in that The historical coordinate data indicates that the target video image detected in the last response to the video image detection request corresponds to the target video image detected in the last response; and determining the drawing area of ​​interest matching the drawing canvas based on the historical coordinate data includes: Determine first coordinate data of the historical coordinate data based on the image zoom ratio; wherein the first coordinate data indicates coordinate data determined with the origin of the coordinate system corresponding to the target container as the vertex; Based on the first coordinate data, determine a coordinate offset; wherein the coordinate offset indicates a minimum value of the first coordinate data on each coordinate axis in a coordinate system corresponding to the target container; Based on the first coordinate data and the coordinate offset, second coordinate data are determined, and based on the second coordinate data, the drawing area of ​​interest is determined; wherein the second coordinate data indicates the coordinate data of the coordinates of the drawing area of ​​interest determined last in response to the video image detection request under the target container.

4. The method according to claim 1, characterized in that: The method further comprises: When it is determined that the vertex coordinate data set does not contain historical coordinate data, an initial drawing area is created; wherein the initial drawing area indicates a central area of ​​the target video image; The interesting drawing area is determined based on the initial drawing area.

5. The method according to claim 4, characterized in that The determining the interested drawing area based on the initial drawing area includes: In response to a mouse trigger operation, determining third coordinate data corresponding to the mouse trigger operation; wherein the third coordinate data indicates the coordinates of the vertex determined by the mouse trigger operation; The interesting drawing region is determined based on the third coordinate data.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: In response to a video stream acquisition request, acquiring a new real-time video stream; wherein the video stream acquisition request is used to request to reacquire the real-time video stream; The target video image is switched to display the frame image of the new real-time video stream, and the frame image of the new real-time video stream is processed to obtain a new video image processing result.

7. The method according to claim 6, characterized in that The method further comprises: Before displaying the frame image of the new real-time video stream, determining whether the resolution information of the new real-time video stream has changed; When it is determined that the resolution information has changed, the frame image of the new real-time video stream is resized, and the frame image of the new real-time video stream is displayed after the resize.

8. A video image processing device, characterized in that: The device comprises: A display unit, configured to display a target video image in response to a video image detection request; wherein the video image detection request is used to request detection of an area of ​​interest of the target video image; and the target video image indicates a frame image of a real-time video stream; A determination unit, configured to create a drawing canvas, and, if it is determined that the vertex coordinate data set includes historical coordinate data, determine a drawing region of interest that matches the drawing canvas based on the historical coordinate data; wherein the vertex coordinate data set is used to store the coordinates of the vertices included in the drawing region of interest; and the historical coordinate data indicates the coordinates of the drawing region of interest determined in response to the video image detection request last time; A processing unit is used to perform drawing processing on the target video image in the drawing canvas based on the drawing area of ​​interest to obtain a video image processing result; wherein the video image processing result indicates an area in the target video image corresponding to the drawing area of ​​interest.

9. The device according to claim 8, characterized in that The display unit is used for: In response to the video image detection request, determining a target container; wherein the target container is a container for displaying the target video image; Determining an image zoom ratio based on container size information of the target container and image size information of the frame image of the real-time video stream; Based on the image scaling ratio, the frame image of the real-time video stream is resized to obtain the target video image, and the target video image is displayed in the target container.

10. The device according to claim 9, characterized in that The historical coordinate data indicates that the target video image detected in response to the last video image detection request corresponds to the target video image detected in response to the last video image detection request; the determining unit is used to: Determine first coordinate data of the historical coordinate data based on the image zoom ratio; wherein the first coordinate data indicates coordinate data determined with the origin of the coordinate system corresponding to the target container as the vertex; Based on the first coordinate data, determine a coordinate offset; wherein the coordinate offset indicates a minimum value of the first coordinate data on each coordinate axis in a coordinate system corresponding to the target container; Based on the first coordinate data and the coordinate offset, second coordinate data are determined, and based on the second coordinate data, the drawing area of ​​interest is determined; wherein the second coordinate data indicates the coordinate data of the coordinates of the drawing area of ​​interest determined last in response to the video image detection request under the target container.

11. The device according to claim 8, characterized in that The device is also used for: When it is determined that the vertex coordinate data set does not contain historical coordinate data, an initial drawing area is created; wherein the initial drawing area indicates a central area of ​​the target video image; The interesting drawing area is determined based on the initial drawing area.

12. The device according to claim 11, characterized in that The device is also used for: In response to a mouse trigger operation, determining third coordinate data corresponding to the mouse trigger operation; wherein the third coordinate data indicates the coordinates of the vertex determined by the mouse trigger operation; The interesting drawing region is determined based on the third coordinate data.

13. The device according to any one of claims 8 to 12, characterized in that The device also includes a video acquisition unit, which is used to: In response to a video stream acquisition request, acquiring a new real-time video stream; wherein the video stream acquisition request is used to request to reacquire the real-time video stream; The target video image is switched to display the frame image of the new real-time video stream, and the frame image of the new real-time video stream is processed to obtain a new video image processing result.

14. The device according to claim 13, characterized in that The device further comprises a resolution detection unit, which is used for: Before displaying the frame image of the new real-time video stream, determining whether the resolution information of the new real-time video stream has changed; When it is determined that the resolution information has changed, the frame image of the new real-time video stream is resized, and the frame image of the new real-time video stream is displayed after the resize.

15. A computer device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the video image processing method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the video image processing method according to any one of claims 1 to 7.

Images