General online AI streaming methods, devices, equipment, and media

Abstract

This application provides a general online AI streaming processing method, apparatus, device, and medium. The method includes: querying the cached results in the video AI buffer of an AI server based on a video prediction request sent by a user; if no cached result corresponding to the URL address is found in the video AI buffer, the video AI processing thread of the AI ​​server checks whether a playback instruction sent by the user has been received within a preset time threshold from the first current time; if a playback instruction sent by the user is confirmed to have been received, the video AI processing thread retrieves the video image from the media server pointed to by the URL address, and sends the AI ​​prediction result of the video image as the cached result corresponding to the URL address to the video AI buffer. This method allows a wide range of internet users to find different media servers based on different URL addresses and perform online AI prediction on the video images from the media servers, which is very convenient.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a general online AI streaming method, apparatus, device, and medium. Background Technology

[0002] Intelligent image processing is a very important and popular area of ​​artificial intelligence (AI), including areas such as face recognition, image classification, and object detection.

[0003] Most current AI models for image processing process single images. Taking face recognition as an example: input an image containing a face, call a pre-trained face recognition AI model to infer the image, and based on the inference result, outline the detected face on the original image and label the identified person's information to obtain the corresponding AI prediction result for the image. However, this method is a single-image-oriented AI processing approach and cannot process video streams. Even if some methods can process video streams, they can only be applied to specific scenarios and specific users.

[0004] Therefore, there is an urgent need to propose an AI streaming processing method for video streams that is applicable to a wide range of internet users. Summary of the Invention

[0005] This application provides a general online AI streaming processing method, apparatus, device, and medium to solve the problem that existing technologies cannot enable any user to perform online AI processing of video streams.

[0006] Firstly, this application provides a general online AI streaming processing method, including:

[0007] Based on the video prediction request sent by the user, the cached results are queried in the video AI buffer of the AI ​​server. The video prediction request includes a URL address, and different URL addresses point to different media servers.

[0008] If no cached result corresponding to the URL address is found in the video AI buffer, the video AI processing thread of the AI ​​server will confirm whether a playback instruction sent by the user terminal has been received within a preset time threshold from the first current time.

[0009] If the playback instruction sent by the user terminal is confirmed to be received, the video AI processing thread obtains the video image from the media server pointed to by the URL address, and sends the AI ​​prediction result of the video image as the cache result corresponding to the URL address to the video AI buffer. The AI ​​prediction result is generated by the image prediction thread of the AI ​​server.

[0010] According to the time difference between the second current time after the AI prediction result is sent to the video AI buffer and the first current time, a frame skipping result is confirmed, and according to the frame skipping result, it is reconfirmed whether the play instruction sent by the user end is received, wherein the frame skipping result includes an instruction of frame skipping and an instruction of no frame skipping.

[0011] In a possible implementation, the confirming, by the video AI processing thread of the AI server, whether the play instruction sent by the user end is received within a preset time threshold from the first current time, comprises:

[0012] The first current time is acquired by the video AI processing thread of the AI server, and the first time at which the user end last sent a play instruction to the media server pointed to by the URL address is acquired;

[0013] If the first difference between the first current time and the first time is less than or equal to the preset time threshold, it is confirmed that the play instruction sent by the user end is received within the preset time threshold of the first current time;

[0014] If the first difference between the first current time and the first time is greater than the preset time threshold, it is confirmed that the play instruction sent by the user end is not received within the preset time threshold of the first current time.

[0015] In a possible implementation, before the AI prediction result of the video image is sent to the video AI buffer as the cache result corresponding to the URL address, the method further comprises:

[0016] According to the video image, the image prediction thread is invoked by the video AI processing thread, and after the AI prediction result of the video image is predicted by the image prediction thread, the AI prediction result is fed back to the video AI processing thread.

[0017] In a possible implementation, the confirming, according to the time difference between the second current time after the AI prediction result is sent to the video AI buffer and the first current time, a frame skipping result, comprises:

[0018] The second current time after the AI prediction result is sent to the video AI buffer is acquired by the video AI processing thread of the AI server, and the second difference between the second current time and the first current time is acquired;

[0019] If the playing duration of the single frame image is greater than or equal to the second difference value, it is confirmed that the frame skipping is not instructed, a third difference value obtained by subtracting the second difference value from the playing duration is taken as the sleep time of the video AI processing thread, and after the sleep is over, it is reconfirmed whether the playing instruction sent by the user end is received;

[0020] If the playing duration of the single frame image is less than the second difference value, it is confirmed that the frame skipping is instructed, the frame skipping processing is performed, and according to the video image pointed to after the frame skipping processing, it is reconfirmed whether the playing instruction sent by the user end is received.

[0021] In a possible implementation, the frame skipping processing is performed, and according to the video image pointed to after the frame skipping processing, it is reconfirmed whether the playing instruction sent by the user end is received, including:

[0022] The second difference value is subtracted from the playing duration to obtain a fourth difference value;

[0023] If the fourth difference value is greater than or equal to the playing duration, the frame skipping is performed once, the fourth difference value is subtracted from the playing duration to obtain an updated fourth difference value, and it is rejudged whether the updated fourth difference value is greater than or equal to the playing duration until the updated fourth difference value is less than the playing duration;

[0024] If the fourth difference value is less than the playing duration, the frame skipping is performed once, and a fifth difference value obtained by subtracting the fourth difference value from the playing duration is taken as the sleep time of the video AI processing thread, and after the sleep is over, it is reconfirmed whether the playing instruction sent by the user end is received.

[0025] In a possible implementation, before the video AI processing thread of the AI server confirms whether the playing instruction sent by the user end is received within the preset time threshold from the first current time, the method further includes:

[0026] The video AI processing thread creates a key-value pair corresponding to the URL address in the video AI buffer area, and the key-value pair will be used to store the cache result corresponding to the URL address;

[0027] If it is confirmed that the playing instruction sent by the user end is not received, the method further includes:

[0028] The key-value pair corresponding to the URL address is deleted in the video AI buffer area, and the related process of the video AI processing thread is ended.

[0029] In a possible implementation, if the cache result corresponding to the URL address is queried in the video AI buffer area, the method further includes:

[0030] record a time when the user terminal sends the playing instruction in the key-value pair corresponding to the URL address, and read the cache result corresponding to the URL address from the key-value pair.

[0031] In a second aspect, the application provides a general online AI streaming processing device, comprising:

[0032] The acquisition module is configured to query a cache result in a video AI buffer of an AI server according to a video prediction request sent by a user terminal, wherein the video prediction request comprises a URL address, and different URL addresses point to different media servers.

[0033] The first processing module is configured to, if the cache result corresponding to the URL address is not queried in the video AI buffer, confirm whether a playing instruction sent by the user terminal is received within a preset time threshold from a first current time through a video AI processing thread of the AI server.

[0034] The second processing module is configured to, if it is confirmed that the playing instruction sent by the user terminal is received, acquire a video image from a media server pointed to by the URL address through the video AI processing thread, and send an AI prediction result of the video image as the cache result corresponding to the URL address to the video AI buffer, wherein the AI prediction result is generated by an image prediction thread of the AI server.

[0035] The frame skipping module is configured to confirm a frame skipping result according to a time difference between a second current time after the AI prediction result is sent to the video AI buffer and the first current time, and reconfirm whether the playing instruction sent by the user terminal is received according to the frame skipping result, wherein the frame skipping result comprises frame skipping and no frame skipping.

[0036] In a third aspect, the application provides a general online AI streaming processing device, comprising at least one processor and a memory.

[0037] The memory stores computer execution instructions.

[0038] The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the general online AI streaming processing method as described above.

[0039] In a fourth aspect, the application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the general online AI streaming processing method as described above.

[0040] The application provides a general online AI streaming processing method, device, equipment and medium. According to a video prediction request sent by a user end, a cache result is inquired in a video AI buffer area of an AI server, wherein the video prediction request comprises a URL address, and different URL addresses point to different media servers. If the cache result corresponding to the URL address is not inquired in the video AI buffer area, whether a play instruction sent by the user end is received within a preset time threshold from a first current time is confirmed through a video AI processing thread of the AI server. If it is confirmed that the play instruction sent by the user end is received, a video image is acquired from the media server pointed to by the URL address through the video AI processing thread, and an AI prediction result of the video image is sent to the video AI buffer area as the cache result corresponding to the URL address, wherein the AI prediction result is generated by an image prediction thread of the AI server. According to a time difference between a second current time after the AI prediction result is sent to the video AI buffer area and the first current time, a frame skipping result is confirmed, and whether the play instruction sent by the user end is received is reconfirmed according to the frame skipping result, wherein the frame skipping result comprises frame skipping and non-frame skipping.

[0041] In the above method, whether the cache result corresponding to the URL address (or the video stream of the media server) exists in the video AI buffer area is inquired through the media server pointed to by the URL address. If not, the video image of the video stream is acquired from the media server through the video AI processing thread, the AI prediction result is generated by sending the video image to the image prediction thread for prediction, and the AI prediction result is sent to the video AI buffer area, so that the AI prediction result can be played for the user. In the process, the processing time of the video AI processing thread is also considered to confirm whether frame skipping is needed when the video image is processed next time, so that online access to the media server by each user is ensured, the playing is smooth, and convenience is provided for the general Internet users. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1 A scene schematic diagram of a general online AI streaming processing provided by the embodiment of the present application;

[0044] Figure 2 A flow schematic diagram of a general online AI streaming processing method provided by the embodiment of the present applicationFigure 1 ;

[0045] Figure 3 A flowchart of a general online AI stream processing method provided for an embodiment of the present application Figure 2 ;

[0046] Figure 4 A flowchart of a general online AI stream processing method provided for an embodiment of the present application Figure 3 ;

[0047] Figure 5 A general online AI stream processing device diagram provided for an embodiment of the present application

[0048] Figure 6 A hardware schematic diagram of a general online AI stream processing device provided for an embodiment of the present application DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0050] Intelligent image processing generally refers to AI model reasoning on an image to complete AI processing of the image. Video stream media is a special form of image. For AI processing of video stream, the AI processing method for processing a single image cannot be directly applied to video stream media. On this basis, the prior art can develop a method for processing video stream: the video is called by taking an offline command line, and each frame of image in the video is intercepted, so that the AI processing of a single image processes each frame of image, and finally the results of AI processing are combined to output to the user as a video. However, such a processing method requires the user to wait, and each user can only operate on a personal terminal device, which cannot provide AI processing services to other users in the form of services, which is very inconvenient.

[0051] Therefore, the present application proposes an AI stream processing method for video stream media, which is more general and can realize online services.

[0052] The following describes how the general online AI stream processing method proposed in the present application is implemented in connection with the drawings and specific embodiments.

[0053] Figure 1This is a schematic diagram illustrating a general online AI streaming processing scenario provided in an embodiment of this application. For example... Figure 1 As shown, the system includes: a user terminal, an AI server, and a media server; wherein, the AI ​​server includes a main module and a service module; the main module includes a video prediction thread and an image prediction thread, and the service module includes a reading thread, a video AI buffer, and a video AI processing thread;

[0054] The client is used to request AI processing services for video streams or images, the AI ​​server is used to provide AI processing services, and the media server is used to provide video streams.

[0055] Each thread has its corresponding processing program. Both the video prediction thread and the image prediction thread can be customized by the user (i.e., the user writes the processing program himself; this application only considers the interaction between the video prediction thread and the basic AI prediction function and interaction between the image prediction thread). After the video AI processing thread is created, the processing program corresponding to the video AI processing thread operates independently.

[0056] The client is used to create video prediction requests (which can correspond to...) Figure 1 In the X0 process (or image prediction request), if a video prediction request (including a URL address and playback request) is generated, the reading thread (which can correspond to the video prediction thread of the main module) is called. Figure 1 The X1 and X2 processes in the video AI buffer allow the reading thread to query the cached result corresponding to the URL address (which can correspond to...) Figure 1 The X3 process in the process; if no cached result is found, a video AI processing thread is created (which can correspond to...). Figure 1 (X4 process in the middle);

[0057] At this point, the video AI processing thread creates key-value pairs corresponding to the URL addresses in the video AI buffer (which can correspond to...). Figure 1 The X5 process (in the original text) retrieves the first current time and obtains the video image from the media server (which can correspond to...). Figure 1 The process from X6 to X8 in the video prediction thread is used to convert the acquired video images into AI prediction results, which are then sent to the video AI buffer (which can correspond to...). Figure 1 (The process from X9 to X12 in the video AI processing thread); After the video AI processing thread sends the AI ​​prediction result of the current frame video image, it uses the time difference between the second current time after the AI ​​prediction result is sent to the video AI buffer and the first current time to determine whether frame skipping is needed, obtains the corresponding frame skipping result, and after knowing the frame skipping result, re-acquires the new first current time and re-predicts the new video image (i.e., after predicting the first frame video image, repeats the process). Figure 1X6 to X13 processes in FIG. 1, continue to predict the following video images).

[0058] In addition to the above processing process, the application can also respond to the image prediction request of the user end through the image prediction thread, and perform AI prediction according to the image prediction request, and feed back the AI prediction result (which can correspond to Figure 1 Y1 to Y3 processes in FIG. 1).

[0059] That is to say, each user can find the corresponding video stream through the URL address on the user end, and the video stream can be controlled online by the video AI processing thread AI prediction processing, that is, continuously obtaining the video stream in the media server corresponding to the URL address, sending the video image in the video stream to the image prediction thread to predict the AI prediction result, and sending the AI prediction result to the video AI buffer; switching different URL addresses can find different media servers, so that each user can continuously obtain the AI prediction result of different video streams according to the play request, and the convenience of AI processing of video stream for the majority of Internet users is provided.

[0060] Figure 2 A general online AI streaming processing method provided by the embodiment of the application Figure 1 . As shown in Figure 2 , the method comprises:

[0061] S201, according to the video prediction request sent by the user end, querying the cache result in the video AI buffer of the AI server, wherein the video prediction request comprises a URL address, and different URL addresses point to different media servers.

[0062] Before the user end sends the video prediction request, a video prediction request needs to be created (which can correspond to the X0 process in FIG. 1); the display interface of the user end comprises an input bar of the URL address, a play key, a stop key and a video playing area; wherein the video playing area can be used to display a single image; Figure 1 The user can input the URL address corresponding to a certain media server in the input bar of the URL address, and click the play key, and then a video prediction request is created and initiated (the video prediction request carries the input URL address), which will be sent to the video prediction thread under the main module of the AI server (which can correspond to the X1 process in FIG. 1); the user can click the stop key at any time after clicking the play key, and the stop key will no longer initiate a video prediction request after being clicked;

[0063] Figure 1 The video prediction thread forwards the video prediction request to the reading thread under the service module of the AI server (which can correspond to the X2 process in FIG. 1).

[0064] The reading thread sends the video prediction request to the image prediction thread under the service module of the AI server (which can correspond to the X3 process in FIG. 1). Figure 1 ​X2 process in FIG. 1) to let the reading thread query whether there is a corresponding cache result in the URL address in the video prediction request in the video AI buffer under the AI server service module;

[0065] If there is, the cache result can be directly fed back: for example, if a video under the URL address (corresponding to the media server) has been watched by other users before a user sends a video prediction request, the video already has a cache result, and the cache result is directly fed back to the user (that is, directly from Figure 1 X3 in FIG. 1 jumps to X14).

[0066] If there is not, a video AI processing thread needs to be created before the cache is started, and the specific processing process is shown in the following steps.

[0067] S202, if the cache result corresponding to the URL address is not queried in the video AI buffer, whether a play instruction sent by the user end is received within a preset time threshold from the first current time is confirmed through the video AI processing thread of the AI server.

[0068] If the cache result corresponding to the URL address is not queried in the video AI buffer, a video AI processing thread is first created (which can correspond to the X4 process in FIG. 1): Figure 1 S201, through the video AI processing thread of the AI server, whether a play instruction sent by the user end is received within a preset time threshold from the first current time is confirmed.

[0069] Before confirming whether the play instruction sent by the user end is received within the preset time threshold from the first current time through the video AI processing thread of the AI server, the method further includes:

[0070] A key-value pair corresponding to the URL address is created in the video AI buffer through the video AI processing thread, and the key-value pair will be used to store the cache result corresponding to the URL address.

[0071] The key-value pair is used to realize a mapping relationship, the key is used as an index, the value is data for storage and reading, the key and the value form a pair of mappings, and the key can index the value corresponding to the key;

[0072] Figure 1 A key-value pair with the URL address as the key is created in the video AI buffer through the video AI processing thread (which can correspond to the X5 process in FIG. 1), so that subsequent processing data is stored in the key-value pair, and the data created for the key-value pair includes a time at which the user end reads a video stream in the media server, a duration for which a frame of video image in the video stream is played, a timeout time for which the video AI processing thread processes the current frame of video image (for subsequent frame skipping processing), and an AI prediction result of the current frame of video image (which can be output as a cache result).

[0073] ​The above process of creating the key-value pair is only created once after the video AI processing thread is created. After completion, the video AI processing thread starts to obtain the first current time when the current progress is reached (which can correspond to Figure 1 the X6 process in the X5 process), and confirms whether a play instruction sent by a user terminal is received within a preset time threshold from the first current time (the time corresponding to the play instruction is recorded in the time at which the user terminal reads the video stream in the media server), where the preset time threshold can be set according to actual engineering requirements.

[0074] If no play instruction is received, it means that there is no user demand for playing, and the example includes:

[0075] If it is confirmed that the play instruction sent by the user terminal is not received, the method further includes:

[0076] deleting the key-value pair corresponding to the URL address in the video AI buffer, and ending the related process of the video AI processing thread.

[0077] In the case where there is no user demand for playing the AI-processed video, the key-value pair corresponding to the URL address can be deleted in the video AI buffer to release data space, and the processing process of the video AI processing thread is ended immediately to release the resources occupied by the computing process, thereby ensuring the smoothness of the entire system.

[0078] If a play instruction is received, it means that there is still a user demand for playing, and further AI processing of the video needs to be started. The specific processing process is shown in the subsequent steps.

[0079] If the cached result corresponding to the URL address is queried in the video AI buffer, the method further includes:

[0080] For example, if the cached result corresponding to the URL address is queried in the video AI buffer, the method further includes:

[0081] recording the time at which the user terminal sends the play instruction in the key-value pair corresponding to the URL address, and reading the cached result corresponding to the URL address from the key-value pair.

[0082] The fact that the cached result corresponding to the URL address can be queried in the video AI buffer indicates that, before the user terminal requests the video in the media server corresponding to the URL address, another user is already playing, and a key-value pair has been created. Therefore, only the corresponding cached result needs to be queried in the key-value pair. In addition, the time at which the user terminal sends the play instruction (i.e., the time at which the user terminal reads the video stream in the media server) also needs to be recorded in the key-value pair for comparison with the first current time.

[0083] S203, if the playing instruction sent by the user terminal is confirmed to be received, the video AI processing thread acquires a video image from the media server pointed by the URL address, and sends the AI prediction result of the video image to the video AI buffer as the cache result corresponding to the URL address, wherein the AI prediction result is generated by the image prediction thread of the AI server.

[0084] If the playing instruction sent by the user terminal is received within the preset time threshold from the first current time, it means that there is still a user requiring to play the video, and the AI processing is performed on the video.

[0085] The URL address input by the user can point to a video of a certain media server, and the video AI processing thread reads a frame of video image from the media server (which can correspond to the X7 process in Figure 1 The media server feeds back the read frame of video image to the video AI processing thread (which can correspond to the X8 process in Figure 1

[0086] The video AI processing thread sends the read frame of video image to the image prediction thread for processing:

[0087] For example, according to the video image, the image prediction thread is called by the video AI processing thread, and after the AI prediction result of the video image is predicted by the image prediction thread, the AI prediction result is fed back to the video AI processing thread.

[0088] The video AI processing thread sends the read frame of video image to the image prediction thread (which can correspond to the X9 process in Figure 1 The image prediction thread performs AI prediction processing on the frame of video image (which can correspond to the X10 process in Figure 1 and obtains the AI prediction result of the frame of video image (for example, AI processing of a real person into a cartoon image, labeling a recognition box for a task, and specific content can be defined by the user terminal), and feeds back the AI prediction result to the video AI processing thread (which can correspond to the X11 process in Figure 1

[0089] The video AI processing thread transmits the AI prediction result as the cache result corresponding to the URL address to the video AI buffer (stored in the AI prediction result data bit of the current frame of video image corresponding to the key-value pair of the URL address) in real time, so that the reading thread can read the cache result.

[0090] ​​S204, according to the time difference between the second current time after the AI prediction result is sent to the video AI buffer and the first current time, confirming the frame skipping result, and according to the frame skipping result, reconfirming whether the play instruction sent by the user end is received, wherein the frame skipping result includes an instruction indicating frame skipping and an instruction indicating no frame skipping.

[0091] When the video AI processing thread is just created, the first frame video image obtained does not need to consider frame skipping processing, but if the AI prediction time of the video AI processing thread does not conform to the playing time under normal playing of a single frame image, then whether frame skipping processing needs to be performed next (which can correspond to the X13 process in Figure 1

[0092] obtaining the second current time after the video AI processing thread sends to the video AI buffer, the time difference between the second current time and the first current time being the AI prediction time of the video AI processing thread, obtaining the frame skipping result according to the result that the AI prediction time is too fast or too slow compared with the playing time under normal playing of a single frame image, and according to the instruction of the frame skipping result, reobtaining the first current time to process the next frame (no frame skipping) video image or frame skipping video image (whether frame skipping or not, the video AI processing thread returns to the X6 process in Figure 3

[0093] In the embodiments of the present application, through the media server pointed by the URL address, it is inquired in the video AI buffer whether there is a cache result corresponding to the URL address (or the video stream of the media server). If not, the video AI processing thread obtains the video image of the video stream from the media server, sends the video image to the image prediction thread to predict the AI prediction result, and sends the AI prediction result to the video AI buffer, so as to play the AI prediction result for the user, and in this process, the processing time of the video AI processing thread is also considered to confirm whether frame skipping is needed when processing the video image next time, so as to ensure that each user can access the media server online, and ensure smooth playing, and provide convenience for the majority of Internet users.

[0094] Figure 2 A flowchart of a general online AI streaming processing method provided by the embodiments of the present application is shown in Figure 3 . As shown in Figure 1 , the method comprises:

[0095] S301, obtaining the first current time through the video AI processing thread of the AI server, and obtaining the first time of the last time when the user end sends a play instruction to the media server pointed by the URL address.​​

[0096] This embodiment corresponds to Figure 4 The specific process of the X6 process is as follows: In order to determine whether the video AI processing thread continues to obtain the video image from the media server pointed to by the URL address, and to provide feedback on the AI ​​prediction result, the video AI processing thread will obtain the first current time when the process reaches the point where it needs to obtain the video image, and read the time data bits of the video stream in the media server from the user terminal in the key-value pair to obtain the first time when the user terminal last sent a playback instruction to the media server pointed to by the URL address.

[0097] S302. If the first difference between the first current time and the first time is less than or equal to the preset time threshold, then it is confirmed that the playback instruction sent by the user terminal is received within the preset time threshold of the first current time.

[0098] The first difference between the first current time and the first time indicates the time difference between the first time when the user last requested to play the video and the first current time. If the first difference is less than or equal to the preset time threshold, it means that the user has issued a playback instruction within the preset time threshold. The user still needs to play the video and needs to continue the X6 to X13 process. For example, if the preset time threshold is 10 seconds and the first difference is 8 seconds, then continue the X6 to X13 process.

[0099] S303. If the first difference between the first current time and the first time is greater than the preset time threshold, then it is confirmed that no playback instruction sent by the user terminal has been received within the preset time threshold of the first current time.

[0100] If the first difference is greater than the preset time threshold, it means that the user has not issued a playback instruction within the preset time threshold, and no user needs to play the video anymore. The relevant processes of the video AI processing thread can be stopped. At this time, the video AI processing thread is deleted, and the X4 to X13 processes related to the video AI processing thread are ended. For example, if the preset time threshold is 10 seconds and the first difference is 12 seconds, the video AI processing thread is deleted, and the X4 to X13 processes related to the video AI processing thread are ended. When the user needs it again, the video AI processing thread can be recreated according to the process of creating the video AI processing thread in the above embodiment.

[0101] In this embodiment, by obtaining the first current time and the first time when the user terminal last sent a playback instruction to the media server pointed to by the URL address, and comparing the time difference between the first current time and the first time with a preset time threshold, it is confirmed whether the playback instruction sent by the user terminal has been received within the preset time threshold from the first current time. In this way, the termination of the video AI processing thread can be controlled to ensure the timely release of resources.

[0102] Figure 3 A flowchart of a general online AI stream processing method provided by an embodiment of the present application Figure 4 As shown in the flowchart, the method comprises the following steps. Figure 1

[0103] S401, obtaining, by a video AI processing thread of the AI server, a second current time after the AI prediction result is sent to the video AI buffer, and obtaining a second difference between the second current time and the first current time.

[0104] This embodiment is a frame skipping process, which corresponds to the specific process of X13 in the process. Figure 1 After the video AI processing thread sends the AI prediction result to the video AI buffer, the second current time is obtained; the second current time is subtracted from the first current time to obtain the second difference, which represents the AI prediction time of the video AI processing thread for AI processing of a single frame of video image (the AI prediction time is recorded in milliseconds, which is very fast for the human eye).

[0105] S402, if the playing duration of the single frame of image is greater than or equal to the second difference, it is confirmed that the frame skipping is not indicated.

[0106] If the playing duration of the single frame of image is greater than or equal to the second difference, it means that the AI prediction time of the video AI processing thread for AI processing of a single frame of video image is too fast, which has exceeded the playing duration of the single frame of image. When playing the single frame of image for the user end, the video AI processing thread has already had idle processing of the next frame of video image, and there is no need to perform frame skipping processing. However, the video AI processing thread will not start processing the next frame of video image immediately.

[0107] S403, taking a third difference obtained by subtracting the playing duration from the second difference as the sleep time of the video AI processing thread, and after the sleep is over, reconfirming whether the playing instruction sent by the user end is received.

[0108] When the AI prediction time is too fast, the video AI processing thread can be put to sleep to ensure that the current frame of image is played smoothly before entering the next processing; the third difference obtained by subtracting the second difference from the playing duration of the single frame of image is the sleep time of the video AI processing thread, which can ensure that the current frame of image is played smoothly. Figure 1 After the sleep is over, the process can return to X6 in the process again to continue obtaining the first current time and reconfirming whether there is a playing instruction, and the first current time is updated to the current time after the sleep is over.

[0109] ​S404, if the playing duration of the single frame image is less than the second difference value, it is confirmed that frame skipping is indicated, and the second difference value and the playing duration are subtracted to obtain a fourth difference value.

[0110] If the playing duration of the single frame image is less than the second difference value, it means that the AI prediction time of the video AI processing thread for AI processing of the single frame video image is too slow, and when the next frame image of the single frame image is played to the user end, the video AI processing thread has not processed the next frame image. In order to ensure that the user looks at the video without lag and more continuous, frame skipping processing can be performed; after confirming that frame skipping is needed, the second difference value and the playing duration are subtracted to obtain a fourth difference value, which clearly indicates how much slower the AI prediction time is than the playing duration of the single frame image.

[0111] S405, if the fourth difference value is greater than or equal to the playing duration, frame skipping is performed once, and the fourth difference value and the playing duration are subtracted.

[0112] If the fourth difference value is greater than or equal to the playing duration, it means that at least once, so first frame skipping processing is performed, and the next frame image is not processed; then the fourth difference value and the playing duration are subtracted.

[0113] For example, the playing duration of the single frame image is 30 milliseconds, the AI prediction time (second difference value) of the first frame image is 40 milliseconds, and the second difference value minus the playing duration of the single frame image is 10 milliseconds. Then there is no time to process the second frame image, so the second frame image is skipped and it is seen whether the third frame image can be processed.

[0114] S406, obtain an updated fourth difference value, and re-determine whether the updated fourth difference value is greater than or equal to the playing duration until the updated fourth difference value is less than the playing duration.

[0115] After the fourth difference value and the playing duration are subtracted, an updated fourth difference value is obtained, and the updated fourth difference value is compared with the playing duration again. If it is still greater than or equal to the playing duration, frame skipping processing is performed again.

[0116] For example, the playing duration of the single frame image is 30 milliseconds, the AI prediction time (second difference value) of the first frame image is 70 milliseconds, and the second difference value minus the playing duration of the single frame image is 40 milliseconds. Then there is no time to process the second frame image, so the second frame image is skipped; and it is returned to S405, 40 milliseconds is compared with 30 milliseconds, and it is found that the updated fourth difference value is still greater than the playing duration of the single frame image, so frame skipping is performed again, the third frame image is skipped, and it is seen whether the fourth frame image can be processed until the updated fourth difference value is less than the playing duration.

[0117] S407. If the fourth difference is less than the playback duration, then perform a frame skip.

[0118] If the fourth difference (or the fourth difference after that) is less than the playback duration, it means that only one more frame skip is needed to catch up with the progress of the video image, so perform the final frame skip.

[0119] Following the example of S406, after skipping the third frame, subtract 30 milliseconds from 40 milliseconds. The updated fourth difference is 10 milliseconds. Since 10 milliseconds is less than 30 milliseconds, skip the frame once more and finally process the fifth frame.

[0120] S408. The fifth difference obtained by subtracting the playback duration from the fourth difference is used as the sleep time of the video AI processing thread. After the sleep ends, it is reconfirmed whether the playback instruction sent by the user terminal has been received.

[0121] If the fourth difference is less than the playback duration, it indicates that a gap has been created after the frame skip. Subtracting the fourth difference from the playback duration yields the fifth difference, which is the gap. The video AI processing thread uses this gap as its sleep time and resumes operation after the sleep period ends. Figure 1 In the X6 process, the first current time is obtained again to reconfirm whether there is a playback instruction. At this time, the first current time is updated to the current time after the hibernation ends.

[0122] For example, the playback duration of a single frame image is 30 milliseconds, the AI ​​prediction time (second difference) of the first frame image is 35 milliseconds, the second difference minus the playback duration of the single frame image is 5 milliseconds, the second frame image needs to be skipped, but before starting to process the third frame image, it is necessary to sleep for 30-5=25 milliseconds.

[0123] In this embodiment, the AI ​​prediction process is reasonably controlled through detailed frame skipping processing to ensure that users have a good online playback experience.

[0124] After AI prediction and cached results are obtained, during user playback, the reading thread feeds back the cached results (AI prediction results) through the video prediction thread. The video playback area on the user's display then plays these AI prediction results (which can correspond to...). Figure 5 The X17 process allows users to pause playback at any time; the reading thread can continuously access the video AI buffer, or the video AI processing thread can sleep for a period of time before accessing the video AI buffer.

[0125] Figure 5 A diagram of a general online AI streaming processing device provided in an embodiment of the present invention is shown, such as... Figure 6As shown, the device comprises: an acquisition module 501, a first processing module 502, a second processing module 503, and a frame skipping module 504.

[0126] The acquisition module 501 is configured to query a cache result in a video AI buffer of an AI server according to a video prediction request sent by a user end, wherein the video prediction request comprises a URL address, and different URL addresses point to different media servers.

[0127] The first processing module 502 is configured to, if the cache result corresponding to the URL address is not queried in the video AI buffer, confirm whether a play instruction sent by the user end is received within a preset time threshold from a first current time through a video AI processing thread of the AI server.

[0128] The first processing module 502 is further configured to acquire the first current time through the video AI processing thread of the AI server, and acquire a first time at which the user end last sent a play instruction to a media server pointed to by the URL address.

[0129] If a first difference between the first current time and the first time is less than or equal to the preset time threshold, it is confirmed that the play instruction sent by the user end is received within the preset time threshold of the first current time.

[0130] If the first difference between the first current time and the first time is greater than the preset time threshold, it is confirmed that the play instruction sent by the user end is not received within the preset time threshold of the first current time.

[0131] The second processing module 503 is configured to, if it is confirmed that the play instruction sent by the user end is received, acquire a video image from the media server pointed to by the URL address through the video AI processing thread, and send an AI prediction result of the video image as the cache result corresponding to the URL address to the video AI buffer, wherein the AI prediction result is generated by an image prediction thread of the AI server.

[0132] The frame skipping module 504 is configured to confirm a frame skipping result according to a time difference between a second current time after the AI prediction result is sent to the video AI buffer and the first current time, and reconfirm whether the play instruction sent by the user end is received according to the frame skipping result, wherein the frame skipping result comprises frame skipping and no frame skipping.

[0133] The frame skipping module 504 is further configured to acquire the second current time after the AI prediction result is sent to the video AI buffer through the video AI processing thread of the AI server, and acquire a second difference between the second current time and the first current time.

[0134] If the playing duration of the single frame image is greater than or equal to the second difference value, it is confirmed that the frame skipping is not instructed, and a third difference value obtained by subtracting the second difference value from the playing duration is taken as the sleep time of the video AI processing thread, and after the sleep is over, it is reconfirmed whether the playing instruction sent by the user end is received;

[0135] If the playing duration of the single frame image is less than the second difference value, it is confirmed that the frame skipping is instructed, and the frame skipping processing is performed, and according to the video image pointed to after the frame skipping processing, it is reconfirmed whether the playing instruction sent by the user end is received.

[0136] The frame skipping module 504 is further configured to subtract the second difference value from the playing duration to obtain a fourth difference value.

[0137] If the fourth difference value is greater than or equal to the playing duration, the frame skipping is performed once, the fourth difference value is subtracted from the playing duration to obtain an updated fourth difference value, and it is rejudged whether the updated fourth difference value is greater than or equal to the playing duration until the updated fourth difference value is less than the playing duration.

[0138] If the fourth difference value is less than the playing duration, the frame skipping is performed once, and a fifth difference value obtained by subtracting the fourth difference value from the playing duration is taken as the sleep time of the video AI processing thread, and after the sleep is over, it is reconfirmed whether the playing instruction sent by the user end is received.

[0139] The application also provides a general online AI streaming processing device, comprising at least one processor and a memory.

[0140] The memory stores computer execution instructions.

[0141] The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the general online AI streaming processing method.

[0142] Figure 6 A hardware schematic diagram of the general online AI streaming processing device provided by the embodiment of the application is shown in the figure. Figure 6 As shown in the figure, the general online AI streaming processing device 60 provided by the embodiment comprises at least one processor 601 and a memory 602. The device 60 further comprises a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected through a bus 604.

[0143] In the specific implementation process, the at least one processor 601 executes the computer execution instructions stored in the memory 602, so that the at least one processor 601 executes the general online AI streaming processing method as above.

[0144] The specific implementation process of the processor 601 can refer to the above-mentioned method embodiments, which have similar implementation principles and technical effects, and details are not described here.

[0145] In the above-mentioned ​ In the embodiment shown, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, CPU for short), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, DSP for short), application specific integrated circuits (English: Application Specific Integrated Circuit, ASIC for short), etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0146] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0147] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus.

[0148] The present application also provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the general online AI streaming processing method is realized.

[0149] The computer readable storage medium described above can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0150] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0151] The division of the units is only a logical function division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0153] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0154] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0155] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0156] Finally, it should be noted that: those skilled in the art will easily think of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application that follow the general principles of the present application and include common knowledge or conventional technical means in the art that are not disclosed in the present application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A general-purpose online AI streaming method, characterized in that, The method comprises the following steps: According to the video prediction request sent by the user terminal, query the cache result in the video AI buffer of the AI server, wherein the video prediction request includes a URL address, and different URL addresses point to different media servers; If no cache result corresponding to the URL address is found in the video AI buffer, confirm whether the play instruction sent by the user terminal is received within a preset time threshold from the first current time through the video AI processing thread of the AI server; If it is confirmed that the play instruction sent by the user terminal is received, acquire the video image from the media server pointed to by the URL address through the video AI processing thread, and send the AI prediction result of the video image as the cache result corresponding to the URL address to the video AI buffer, wherein the AI prediction result is generated by the image prediction thread of the AI server; According to the time difference between the second current time after sending the AI prediction result to the video AI buffer and the first current time, confirm the frame skipping result, and according to the frame skipping result, reconfirm whether the play instruction sent by the user terminal is received, wherein the frame skipping result includes instruction to skip frame and instruction not to skip frame; Before the step of confirming whether the play instruction sent by the user terminal is received within a preset time threshold from the first current time through the video AI processing thread of the AI server, the method further comprises the following steps: Create a key-value pair corresponding to the URL address in the video AI buffer through the video AI processing thread, and the key-value pair will be used to store the cache result corresponding to the URL address; after creating the key-value pair corresponding to the URL address in the video AI buffer through the video AI processing thread, acquire the first current time.

2. The method of claim 1, wherein, The step of confirming whether the play instruction sent by the user terminal is received within a preset time threshold from the first current time through the video AI processing thread of the AI server comprises the following steps: Acquire the first current time through the video AI processing thread of the AI server, and acquire the first time when the user terminal last sent the play instruction to the media server pointed to by the URL address; If the first difference between the first current time and the first time is less than or equal to the preset time threshold, it is confirmed that the play instruction sent by the user terminal is received within the preset time threshold of the first current time; If the first difference between the first current time and the first time is greater than the preset time threshold, it is confirmed that the play instruction sent by the user terminal is not received within the preset time threshold of the first current time.

3. The method of claim 1, wherein, Before the step of sending the AI prediction result of the video image as the cache result corresponding to the URL address to the video AI buffer, the method further comprises the following steps: According to the video image, call the image prediction thread through the video AI processing thread, and after the image prediction thread predicts the AI prediction result from the video image, feed back the AI prediction result to the video AI processing thread.

4. The method of claim 1, wherein, The second current time after the AI prediction result is sent to the video AI buffer is obtained by a video AI processing thread of the AI server, and a second difference value between the second current time and the first current time is obtained. If the playing duration of a single frame image is greater than or equal to the second difference value, it is determined that frame skipping is not indicated, and a third difference value obtained by subtracting the second difference value from the playing duration is taken as a sleep time of the video AI processing thread, and after the sleep is over, it is determined again whether the playing instruction sent by the user end is received. If the playing duration of a single frame image is less than the second difference value, it is determined that frame skipping is indicated, and frame skipping processing is performed, and according to a video image pointed to after the frame skipping processing, it is determined again whether the playing instruction sent by the user end is received. The frame skipping processing includes:

5. The method of claim 4, wherein, The second difference value is subtracted from the playing duration to obtain a fourth difference value. If the fourth difference value is greater than or equal to the playing duration, frame skipping is performed once, the fourth difference value is subtracted from the playing duration to obtain an updated fourth difference value, and it is determined again whether the updated fourth difference value is greater than or equal to the playing duration until the updated fourth difference value is less than the playing duration. If the fourth difference value is less than the playing duration, frame skipping is performed once, and a fifth difference value obtained by subtracting the fourth difference value from the playing duration is taken as a sleep time of the video AI processing thread, and after the sleep is over, it is determined again whether the playing instruction sent by the user end is received.

6. The method of claim 1, wherein If it is determined that the playing instruction sent by the user end is not received, the method further includes: deleting the key-value pair corresponding to the URL address in the video AI buffer, and ending the related process of the video AI processing thread. If the cache result corresponding to the URL address is queried in the video AI buffer, the method further includes:

7. The method of claim 6, wherein, recording the time when the playing instruction is sent by the user end in the key-value pair corresponding to the URL address, and reading the cache result corresponding to the URL address from the key-value pair. The method includes:

8. A general-purpose online AI streaming device, characterized by, an obtaining module configured to query a cache result in a video AI buffer of an AI server according to a video prediction request sent by a user end, wherein the video prediction request includes a URL address, and different URL addresses point to different media servers; a first processing module configured to, if the cache result corresponding to the URL address is not queried in the video AI buffer, determine, by a video AI processing thread of the AI server, whether a playing instruction sent by the user end is received within a preset time threshold from a first current time; ​ The second processing module is configured to, if it is confirmed that the playing instruction sent by the user terminal is received, acquire a video image from a media server pointed by the URL address through the video AI processing thread, and send an AI prediction result of the video image as a cache result corresponding to the URL address to the video AI buffer, wherein the AI prediction result is generated by an image prediction thread of the AI server. The frame skipping module is configured to, according to a time difference between a second current time after the AI prediction result is sent to the video AI buffer and the first current time, confirm a frame skipping result, and according to the frame skipping result, reconfirm whether the playing instruction sent by the user terminal is received, wherein the frame skipping result includes frame skipping and no frame skipping. The first processing module is further configured to, before confirming whether the playing instruction sent by the user terminal is received within a preset time threshold from the first current time through the video AI processing thread of the AI server, create a key-value pair corresponding to the URL address in the video AI buffer through the video AI processing thread, and the key-value pair is used to store the cache result corresponding to the URL address; and the video AI processing thread acquires the first current time after creating the key-value pair corresponding to the URL address in the video AI buffer.

9. A general-purpose online AI stream processing device, characterized by, The computer program is executed by the processor to implement the steps of the general online AI streaming processing method according to any one of claims 1-7. The computer program is executed by the processor to implement the steps of the general online AI streaming processing method according to any one of claims 1-7. ​ ​ 10. A computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images