Artificial intelligence inference method on video stream, computing device and edge device

Abstract

This invention proposes an artificial intelligence (AI) inference method, a computing device, and an edge device for video streams. The aforementioned computing device includes a first processor and a second processor. The first processor is configured to continuously receive a main video stream. The second processor is configured to perform scaling processing on the main video stream to generate a scaled video stream, and to perform AI inference on the first frame of the scaled video stream to generate an AI inference result for the second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the AI ​​inference result is generated or the time point after generation.

Description

Technical Field

[0001] This invention relates to an artificial intelligence (AI) inference technology for video streaming. Background Technology

[0002] Edge devices refer to devices such as sensors, gateways, actuators, and IoT devices that can collect and process data at the network edge. This edge computing architecture with AI inference not only brings computation closer to the data source, significantly reducing the need for large-scale data transmission to the cloud, but also saves network bandwidth, enables faster decision-making, and reduces reaction time. However, with the development of high-quality video streaming and high-frame-rate display technologies, performing real-time AI inference at the edge will face challenges given limited power and computing resources. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides an AI inference method for video streams, along with its computing device and edge device.

[0004] In one embodiment of the present invention, the aforementioned computing device includes a first processor and a second processor. The first processor is configured to continuously receive a main video stream. The second processor is configured to perform scaling processing on the main video stream to generate a scaled video stream, and to perform AI inference on a first frame of the scaled video stream to generate an AI inference result for a second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the AI ​​inference result is generated or the time point after generation.

[0005] In one embodiment of the present invention, the aforementioned computing device includes a first processor, a second processor, and a screen display controller. The first processor is configured to continuously receive a main video stream. The second processor is configured to perform scaling processing on the main video stream to generate a scaled video stream, and to perform AI inference on a first frame of the scaled video stream to generate an AI inference result. The screen display controller is configured to overlay text or graphics onto a second frame of the main video stream based on the AI ​​inference result, wherein the second frame of the main video stream is the frame corresponding to the time point when the AI ​​inference result is generated or the time point after generation.

[0006] In one embodiment of the present invention, the aforementioned edge device includes a computing unit and a screen display. The computing unit includes a first processor, a second processor, and a screen display controller. The first processor is configured to continuously receive a main video stream. The second processor is configured to perform scaling processing on the main video stream to generate a scaled video stream, and to perform AI inference on a first frame of the scaled video stream to generate an AI inference result. The screen display controller is configured to overlay text or graphics onto a second frame of the main video stream based on the AI ​​inference result to generate a processed second frame, wherein the second frame of the main video stream corresponds to the time point at which the AI ​​inference result is generated or to a time point after the generation. The screen display is configured to display the processed second frame.

[0007] In one embodiment of the present invention, the above method includes continuously receiving a main video stream, scaling the main video stream to generate a scaled video stream, and performing AI inference on a first frame of the scaled video stream to generate an AI inference result for a second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the AI ​​inference result is generated or the time point after generation.

[0008] In one embodiment of the present invention, the above method includes continuously receiving a main video stream, scaling the main video stream to generate a scaled video stream, performing AI inference on the first frame of the scaled video stream to generate an AI inference result, and superimposing text or graphics on the second frame of the main video stream according to the AI ​​inference result, wherein the second frame of the main video stream is the frame corresponding to the time point when the AI ​​inference result is generated or the time point after the generation.

[0009] In one embodiment of the present invention, the method includes continuously receiving a main video stream, performing scaling processing on the main video stream to generate a scaled video stream, performing AI inference on the first frame of the scaled video stream to generate an AI inference result, superimposing text or graphics on the second frame of the main video stream according to the AI ​​inference result to generate a processed second frame, and displaying the processed second frame on a screen display, wherein the second frame of the main video stream is the frame corresponding to the time point when the AI ​​inference result is generated or the time point after the generation. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a computing device according to an embodiment of the present invention.

[0011] Figure 2 This is a flowchart illustrating a method for performing AI inference on a video stream according to an embodiment of the present invention.

[0012] Figure 3 This is an adaptive AI inference architecture illustrated according to an embodiment of the present invention.

[0013] Figure 4 This is a schematic diagram of an edge device according to an embodiment of the present invention.

[0014] Figure 5 This is a flowchart illustrating a method for performing AI inference on a video stream according to an embodiment of the present invention.

[0015] Figure 6 This is a schematic diagram illustrating how an edge device operates according to an embodiment of the present invention. Detailed Implementation

[0016] Some exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The number of elements appearing in the following drawings is for illustrative purposes only, and the present invention is not intended to be limited thereto. Furthermore, these embodiments are only a part of the present invention and do not disclose all possible implementations of the present invention. More precisely, these embodiments are merely examples of methods and computer systems within the scope of the present invention's patent application.

[0017] Figure 1 This is a schematic diagram of a computing device according to an embodiment of the present invention. First Figure 1 The various components and their configuration relationships in the computing device will be introduced first. Detailed functions will be explained in conjunction with the flow of subsequent embodiments. Figure 1 And made public.

[0018] Please refer to Figure 1 The computing device 100 of this embodiment includes a first processor 110 and a second processor 120 coupled to or connected thereto. The computing device 110 may be a stand-alone computer or a hardware architecture embedded in an edge device with image processing capabilities. For example, the edge device may be an in-vehicle computer capable of alerting potential road hazards. The first processor 110 and the second processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a programmable general-purpose or special-purpose microprocessor (microprocessor), a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), other similar devices, integrated circuits, or combinations thereof. The first processor 110 and the second processor 120 may also be integrated circuits, such as a system-on-a-chip (SoC), but the invention is not limited thereto.

[0019] Figure 2 This is a flowchart illustrating a method for performing AI inference on a video stream according to an embodiment of the present invention, wherein... Figure 2 The steps can be made by Figure 1 The shown computing device 100 is implemented.

[0020] Please refer to the following at the same time Figure 2 as well as Figure 1 The first processor 110 of the computing device 100 continuously receives the main video stream (step S202). In this embodiment, the main video stream may be a live video feed received from a specific source, such as a vehicle camera, surveillance camera, or any other video source device. In other embodiments, the main video stream may also be an offline video stream, such as a game, video animation, or pre-stored video content.

[0021] Next, the second processor 120 of the computing device 100 performs scaling processing on the main video stream to generate a scaled video stream (step S204). In computer graphics, image scaling (also known as image resizing) mainly includes image reduction and image magnification. Image reduction reduces the original image data according to an image reduction ratio to reduce the computational burden and algorithm execution time. Image magnification magnifies the original image data according to an image magnification ratio to examine details in local areas. In one scenario, the second processor 120 may perform image reduction processing on the main video stream to generate a scaled video stream. In another scenario, the second processor 120 may perform image magnification processing on a predetermined region in the main video stream that has potential or confirmed specific features to generate a scaled video stream. In yet another scenario, the second processor 120 may perform image reduction processing on the aforementioned predetermined region of the main video stream to obtain optimized processing efficiency.

[0022] The second processor 120 can perform various video analysis processes on the scaled video stream according to any AI inference method, such as object detection, scene recognition, and face recognition, to make predictive decisions. To smoothly process video images and achieve low-latency AI inference for real-time response under limited computing resources, the second processor 120 can perform AI inference on the scaled video stream at dynamic inference time points based on image content. Specifically, the second processor 120 of the computing device 100 performs AI inference on the first frame of the scaled video stream to generate the AI ​​inference result for the second frame of the main video stream (step S206), where the second frame of the main video stream refers to the frame corresponding to the time point when the AI ​​inference result is generated or the time point after the generation. For example, the second frame of the main video stream can be the current frame in the main video stream at the time point when the AI ​​inference result is generated, or the next frame in the main video stream after the generation of the AI ​​inference result.

[0023] It should be noted that since the time span of each AI inference changes dynamically based on the complexity of the image content, the next AI inference execution will be performed on the frame corresponding to the latest frame of the main video stream continuously received by the first processor 110 within the scaled video stream. In other words, the second processor 120 will perform AI inference on the frame of the scaled video stream corresponding to the time point when or after the AI ​​inference result is generated, to generate a new AI inference result for the third frame of the main video stream, where the third frame of the main video stream refers to the frame corresponding to the time point when or after the new AI inference result is generated.

[0024] To better understand, Figure 3 This is an adaptive AI inference architecture illustrated according to an embodiment of the present invention.

[0025] Please refer to Figure 3Two aligned time axes T plot the main video stream containing frames F1-F7 and the scaled video stream containing frames f1-f7. The second processor 120 begins performing AI inference INF1 on frame f1 of the scaled video stream after the data enable (DE) signal corresponding to time point t1 becomes inactive, and generates the AI ​​inference result at time point t2. Here, the AI ​​inference result generated at time point t2 is applicable to either frame F3 or frame F4 of the main video stream. Since the time span of the AI ​​inference occurs after the data enable signal corresponding to frame f3 has already become active, the second processor 120 discards frame f2 of the scaled video stream and begins processing frame f3 corresponding to the latest frame F3 of the main video stream. In other words, the second processor 120 begins performing AI inference INF2 on frame f3 after the data enable signal corresponding to time point t3 becomes inactive, and generates the AI ​​inference result at time point t4, which is applicable to either frame F4 or frame F5 of the main video stream. Similarly, the second processor 120 begins performing AI inference INF3 on frame f4 after the data enable signal corresponding to time point t5 becomes inactive, and generates the AI ​​inference result at time point t6. The AI ​​inference result generated at time point t6 is applicable to frame f7 of the main video stream or its subsequent frames. In this case, the time span of AI inference INF1, INF2, and INF3 is dynamic and varies depending on the complexity of the image content in frames f1, f3, and f4. Furthermore, although AI inference is not performed on every frame, this process is difficult for humans to perceive.

[0026] As an application scenario, Figure 4 This is a schematic diagram of an edge device according to an embodiment of the present invention. First Figure 4 The various components and their configuration relationships in the computing device will be introduced first. Detailed functions will be explained in conjunction with the flow of subsequent embodiments. Figure 1 And made public.

[0027] Please refer to Figure 4 The edge device 40 includes a computing unit 400 and a screen display 450. The edge device 40 can be a vehicle-mounted computer, personal computer, mobile device, or Internet of Things (IoT) device (e.g., a smart TV, a smart surveillance camera, etc.). The computing unit 400 also includes a first processor 410, a second processor 420, and a screen display controller 430, wherein the second processor 420 and the screen display controller 430 are coupled to or connected to the first processor 410, and the screen display controller 430 is coupled to or connected to the second processor 420. Here, the hardware configuration of the first processor 410 and the second processor 420 of the computing device 400 is similar to... Figure 2The computing device 200 includes a first processor 210 and a second processor 220. The screen display controller 430 may be a digital circuit that provides screen display creation functions and may be considered as a third processor of the computing device 400.

[0028] Figure 5 This is a flowchart illustrating a method for performing AI inference on a video stream according to an embodiment of the present invention, wherein... Figure 5 The steps can be made by Figure 4 The edge device 40 shown is implemented.

[0029] Please refer to the following at the same time Figure 5 as well as Figure 4 The first processor 410 of the computing unit 400 of the edge device 40 continuously receives the main video stream (step S502), and the second processor 420 of the computing unit 400 of the edge device 40 performs scaling processing on the main video stream to generate a scaled video stream (step S504). AI inference is then performed on the first frame of the scaled video stream to generate an AI inference result (step S506). Those skilled in the art can deduce the details of steps S502-S506 based on steps S202-S206 in step 2, and will not be repeated here.

[0030] In this embodiment, the second processor 420 can perform image analysis on the first frame of the scaled video stream based on any image recognition technology to determine a specified object in the first frame of the scaled video stream. This specified object can be a specific target, a specific area, or even the background scene in the first frame that needs to be detected and monitored. The second processor 420 will then use AI inference results for the specified object in the second frame of the main video stream. As previously mentioned, the second frame refers to the frame corresponding to the time point when the AI ​​inference result is generated or the time point after its generation.

[0031] The AI ​​inference results can be output in various forms and formats. In this embodiment, the AI ​​inference results are text or graphics that can achieve a visual effect. The screen display controller 430 overlays the text or graphics onto the second frame of the main video stream based on the AI ​​inference results to generate a processed second frame (step S508), and the screen display 450 displays the processed second frame (step S510). It should be noted that the text or graphics can be overlaid at a position associated with the aforementioned specified object or in a predetermined area of ​​the second frame of the main video stream.

[0032] For example, Figure 6 This is a schematic diagram illustrating how an edge device operates according to an embodiment of the present invention. In this example embodiment, the edge device 40 may be an in-vehicle monitoring system for enhancing security.

[0033] Please refer to Figure 6Frame 610 in the main video stream includes a white vehicle W and a yellow vehicle Y. Frame 610 is scaled down to frame 620 based on an image downscaling ratio for AI inference. The white vehicle W' and yellow vehicle Y' in frame 620 are identified as potential collision risks by the AI ​​inference, alerting the driver. The positions in frame 620 identified as AI inference results can be mapped to corresponding positions in the next frame 630 of the main video stream based on the image downscaling ratio. In this scenario, the AI ​​inference results will be presented as bordered white vehicle W' and bordered yellow vehicle Y' in the next frame 630 of the main video stream after generation, to alert the driver. These borders may appear in one or more subsequent frames until a new AI inference result is generated.

[0034] Please refer to again Figure 1 In another embodiment, Figure 1 The computing device 100 may also include an audio signal processor or digital circuitry to generate a voice signal associated with a specified object in a second frame of the main video stream, based on AI inference results. For example, the voice signal could be an alarm message output from a speaker to notify the user of the presence of the specified object.

[0035] In another embodiment, Figure 1 The computing device 100 may also include a backlight controller or digital circuitry to generate control signals associated with the display of a second frame of the main video stream based on AI inference results. For example, assuming the AI ​​inference results indicate that the computing device 100 is in a dark environment, the control signals could provide sufficient backlight to view the second frame of the main video stream.

[0036] In summary, the adaptive AI inference architecture proposed in this invention allows edge devices with limited power and computing resources to perform real-time AI inference at the edge without compromising on high-quality video streaming and high-frame-rate hardware.

[0037] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the disclosed embodiments without departing from the scope or spirit of this application. In summary, this application is intended to cover any modifications and variations of this application that fall within the scope of the following claims and their equivalents.

Claims

1. A computing device, comprising: The first processor is configured to continuously receive the main video stream; as well as The second processor is configured to: The main video stream is scaled to generate a scaled video stream; as well as Artificial intelligence inference is performed on the first frame of the scaled video stream to generate an artificial intelligence inference result for the second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the artificial intelligence inference result is generated or the time point after the generation.

2. The computing apparatus of claim 1, wherein the second processor performs image downscaling processing on the main video stream to generate the scaled video stream.

3. The computing device of claim 1, wherein the processor performs image magnification processing on a predetermined region of the main video stream to generate the scaled video stream.

4. The computing apparatus of claim 1, wherein the second frame of the main video stream is the current frame of the main video stream at the time when the artificial intelligence inference result is generated.

5. The computing apparatus of claim 1, wherein the second frame of the main video is the next frame of the main video stream at the time point after the generation of the artificial intelligence inference result.

6. The computing apparatus of claim 1, wherein the second processor performs image analysis on the first frame of the scaled video stream to determine a specified object in the first frame of the scaled video stream, and The second processor generates the artificial intelligence inference result for the specified object in the second frame of the main video stream.

7. The computing device of claim 6, further comprising: The third processor, configured as follows: The text or graphic is generated based on the artificial intelligence inference result, and the position of the text or graphic associated with the specified object is superimposed in the second frame of the main video stream.

8. The computing device of claim 7, wherein the third processor determines the position of the text or the graphic superimposed on the second frame of the main video stream based on the first frame of the scaled video stream and the image scaling ratio of the first frame.

9. The computing device of claim 6, further comprising: The fourth processor is configured to: Based on the artificial intelligence inference results, an audio signal associated with the specified object in the second frame of the main video stream is generated.

10. The computing device of claim 6, further comprising: The fifth processor is configured to: Based on the artificial intelligence inference results, control signals associated with the display of the second frame of the main video stream are generated.

11. The computing device of claim 1, wherein the second processor is further configured to: Artificial intelligence inference is performed on the frames of the scaled video stream corresponding to the time point when the artificial intelligence inference result is generated or the time point after the generation, to generate a new artificial intelligence inference result for the third frame of the main video stream, wherein the third frame of the main video stream is the frame corresponding to the time point when the new artificial intelligence inference result is generated or the time point after the generation.

12. A computing device, comprising: The first processor is configured to continuously receive the main video stream; The second processor is configured to: The main video stream is scaled to generate a scaled video stream; Artificial intelligence inference is performed on the first frame of the scaled video stream to generate an artificial intelligence inference result; as well as The screen display controller is configured to: Based on the artificial intelligence inference result, text or graphics are superimposed on the second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the artificial intelligence inference result is generated or the time point after the generation.

13. The computing apparatus of claim 12, wherein the second processor performs image analysis on the first frame of the scaled video stream to determine a specified object in the first frame of the scaled video stream, and The second processor generates the artificial intelligence inference result for the specified object in the second frame of the main video stream.

14. The computing device as claimed in claim 12, Artificial intelligence inference is performed on the frames of the scaled video stream corresponding to the time point when the artificial intelligence inference result is generated or the time point after the generation, to generate a new artificial intelligence inference result for the third frame of the main video stream, wherein the third frame of the main video stream is the frame corresponding to the time point when the new artificial intelligence inference result is generated or the time point after the generation.

15. An edge device, comprising: A computing device, comprising: The first processor is configured to continuously receive the main video stream; The second processor is configured to: Perform scaling processing on the main video stream to generate a scaled video stream; and Perform AI inference on the first frame of the scaled video stream to generate an AI inference result; and The screen display controller is configured to: Based on the AI ​​inference result, text or graphics are superimposed on the second frame of the main video stream to generate a processed second frame, wherein the second frame of the main video stream corresponds to the time point at which the AI ​​inference result is generated or the time point after its generation; and The screen display is configured to: The processed second frame is displayed.

16. The edge device of claim 15, wherein the second processor performs image analysis on the first frame of the scaled video stream to determine a specified object in the first frame of the scaled video stream. The second processor generates the artificial intelligence inference result for the specified object in the second frame of the main video stream.

17. The edge device of claim 15, performing artificial intelligence inference on frames of the scaled video stream corresponding to the time point at which the artificial intelligence inference result is generated or the time point after the generation, to generate a new artificial intelligence inference result for a third frame of the main video stream, wherein the third frame of the main video stream is the frame corresponding to the time point at which the new artificial intelligence inference result is generated or the time point after the generation.

18. A calculation method, comprising: Continuously receive the main video stream; The main video stream is scaled to generate a scaled video stream; as well as Artificial intelligence inference is performed on the first frame of the scaled video stream to generate an artificial intelligence inference result for the second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the artificial intelligence inference result is generated or the time point after the generation.

19. A calculation method, comprising: Continuously receive the main video stream; The main video stream is scaled to generate a scaled video stream; Perform AI inference on the first frame of the scaled video stream to generate AI inference results; as well as Based on the artificial intelligence inference result, text or graphics are superimposed on the second frame of the main video stream, wherein the second frame of the main video stream is the frame corresponding to the time point when the artificial intelligence inference result is generated or the time point after the generation.

20. A method applicable to an edge device, comprising: Continuously receive the main video stream; The main video stream is scaled to generate a scaled video stream; Perform AI inference on the first frame of the scaled video stream to generate AI inference results; Based on the artificial intelligence inference result, text or graphics are superimposed on the second frame of the main video stream to generate a processed second frame, wherein the second frame of the main video stream is a frame corresponding to the time point when or after the generation of the artificial intelligence inference result; as well as The processed second frame is displayed on the screen.

Images