Framework method based on video edge AI reasoning cluster

Abstract

The invention discloses an architecture method based on a video edge AI inference cluster, and relates to video data processing, the method comprises the following steps: configuring an inference cluster comprising a plurality of edge devices and accessing a server; the server specifies the identity of the edge device and configures a load registry, the main device obtains the load registry through the server and establishes a distribution decision, and the main device obtains a video stream of the data acquisition node and decodes and converts the video stream into a picture format to obtain a plurality of continuous first image data; meanwhile, the master device accesses a load registry and sends the first image data to a corresponding slave device according to a distribution decision, and the slave device calls an inference model to perform an inference service; the method aims to improve the situation that when a current edge device carries out AI reasoning, due to resource limitation of the edge device, resource distribution is uneven or waste is caused.

Description

Technical Field

[0001] The present invention relates to video data processing, and in particular to an architecture method based on a video edge AI inference cluster. Background Art

[0002] With the development of technology, video edge AI reasoning refers to the deployment of artificial intelligence (AI) reasoning service models directly on edge devices close to the data source in the video processing system, so that the system can respond quickly and reduce the computing power consumption of remote servers or clouds by running data models on edge devices. With the diverse process requirements of AI reasoning, the current deployment method of edge devices is easily limited by the resources of edge devices. However, the mainstream of existing edge reasoning is to use a specified method to allocate tasks. When the load of edge devices changes, the system may lack effective processing efficiency. On the one hand, edge devices usually have limited computing power and storage capacity, and it is difficult to handle a large number of concurrent tasks, especially when all edge devices transmit raw data or processing results back to the central server, which will cause a waste of network bandwidth; on the other hand, for a small number of tasks, and continuous video frames may contain a large amount of similar information, it is easy to generate redundancy in the reasoning process, resulting in uneven or wasteful resource allocation; especially for video streams with large data volumes, the original layout method is not conducive to insufficient performance optimization measures, and the reconfiguration cycle of the equipment is long, and the overall flexibility is poor, which makes the application scenario service performance lacking. Therefore, it is worth studying how to use edge devices for video edge AI reasoning. Summary of the invention

[0003] The purpose of the present invention is to provide an architecture method based on a video edge AI inference cluster, in the hope of improving the situation where when the current edge device performs AI inference, there may be uneven resource allocation or waste due to the resource limitations of the edge device itself.

[0004] An embodiment of the present invention provides a method for processing images from a device, by processing and extracting frame images in order to retain the processing of key frame images and reduce unnecessary image processing, thereby reducing the computational complexity and processing time of the inference service.

[0005] An embodiment of the present invention provides a configuration strategy for edge devices, which configures edge devices by setting the health of edge devices in conjunction with a load registry, in the hope that edge devices can be replaced or assisted when executing tasks to cope with the impact of device abnormalities or high-concurrency tasks.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] An architecture method based on a video edge AI inference cluster includes the following steps: S100, configuring an inference cluster, where the inference cluster includes multiple edge devices, and the edge devices are connected to a server; signals are interacted between the edge devices; the server designates an edge device as a master device and designates multiple edge devices as slave devices.

[0008] S200, the server configures a load registry, which includes the identity, load information and priority of each edge device; the main device obtains the load registry through the server and establishes an allocation decision.

[0009] S300, the master device acquires the video stream of the data acquisition node and decodes the video stream into a picture format to obtain a number of continuous first image data; at the same time, the master device accesses the load registry and sends the first image data to the corresponding slave device according to the allocation decision, and the slave device calls the inference model to perform inference services. The video stream is a digital signal of the video captured in real time by the data acquisition node; the first image data includes a single-frame picture of a number of consecutive frames.

[0010] The above-mentioned master device establishing allocation decision includes the following steps:

[0011] S210, the master device downloads the load registry from the server and extracts the load information, and calculates the load health through the load information; S220, sorts the slave devices in the load registry according to the load health to generate a priority table; S230, the master device selects a slave device from the priority table according to a set number to establish an initial connection and obtains the number of task processes of the slave device; the master device determines the allocation order according to the number of task processes; wherein, when the master device establishes an initial connection with the slave device, the master device feeds back the connection status to the server, and the server marks the corresponding slave device in the load registry as working.

[0012] Preferably, it also includes S400, wherein the slave device generates image data from the first image data through an inference service, and the image data is returned to the master device through a load balancing interface. The master device forwards the image data to a server, and the server displays it to a front-end web page through a websocket channel.

[0013] A further technical solution is that when the slave device performs the inference service, the slave device calls the inference model and generates picture data from the first image data through the inference model; the execution steps of the inference service are:

[0014] S410, extracting all single-frame pictures in the first image data, and determining key frames and intermediate frames in the first image data.

[0015] S420: Set the single-frame picture corresponding to the key frame as the key picture, and set the single-frame picture adjacent to the key picture as the verification picture.

[0016] There are multiple key frames extracted from the first image data, and each key frame is set as a key picture.

[0017] S430, calling the built-in preset model from the device to perform data processing on the key image and the verification image respectively, and extracting edge features of the key image and the verification image by the preset model to generate a first edge image and a second edge image respectively.

[0018] The preset model can be any of the VGGNet, ResNet or Inception models;

[0019] S440, the above-mentioned slave device sets a similarity parameter, and judges the similarity between the first edge image of the key image and the second edge image of the corresponding verification image to determine whether the selection of the key frame is normal; when the similarity is greater than or equal to the similarity parameter, it indicates that the selection of the key frame is normal, and the above-mentioned slave device compresses the first edge image to obtain a compressed image, and executes S450; when the similarity is less than the similarity parameter, it indicates that the selection of the key frame is abnormal, and the key frame is adjusted to an intermediate frame.

[0020] S450: Integrate the compressed pictures into picture data, where the picture data includes all compressed pictures corresponding to the first image data.

[0021] A further technical solution is that when executing S410, the above-mentioned slave device sets a cumulative number; obtains the number of all single-frame pictures in the first image data and determines whether the number of single-frame pictures exceeds the cumulative number of pictures; when the number of single-frame pictures exceeds the cumulative number, the above-mentioned slave device executes a preset dynamic strategy, and the slave device divides the first image data into second image data and third image data, and the above-mentioned second image data is used to execute steps S420 to S450.

[0022] The second image data includes single-frame pictures whose number is the same as the accumulated number of pictures.

[0023] The third image data includes a single frame picture of which the first image data exceeds the cumulative number.

[0024] The slave device sends a request to the server, the server selects an idle edge device from the load registry and sets the identity to the assisting device, the slave device forwards the third image data to the assisting device, and the assisting device executes steps S420 to S450.

[0025] A further technical solution is that when executing S440, the similarity parameter set by the slave device is T, and the similarity between the first edge image and the corresponding second edge image is obtained; the calculation formula of the similarity S is:

[0026]

[0027] In the formula, u A is the average brightness of the first edge image; u B is the average brightness of the second edge image; β AB is the covariance between the first edge image and the second edge image; is the variance of the first edge image; is the variance of the second edge picture; L1 and L2 are the stability constants of the first edge picture and the second edge picture respectively.

[0028] When S≥T, the selection of the key frame is considered normal; when S<T, the selection of the key frame is considered abnormal.

[0029] One of the technical solutions is that when the number of second edge pictures is equal to 1, the first edge picture is the start frame or the end frame of the first image data, and the similarity S between the first edge picture and the corresponding second edge picture is directly obtained by the above.

[0030] One of the technical solutions is that when the number of second edge pictures is equal to 2, the second edge pictures are respectively the two frames before and after the first edge picture; the familiarity S1 between the second edge picture of the previous frame and the first edge picture, and the familiarity S2 between the second edge picture of the next frame and the first edge picture are obtained according to the above formula; the similarity S between the first edge picture and the corresponding second edge picture is the average value of S1 and S2.

[0031] Preferably, in S300, when the master device sends the first image data to the slave device for inference, the master device sets a monitoring time, during which the master device continuously monitors the progress of the corresponding slave device's inference and determines whether the processing efficiency of the slave device is normal.

[0032] If the master device receives the image data returned by the slave device within the monitoring time, it means that the processing efficiency of the slave device is normal.

[0033] Among them, if the main device does not receive the image data within the monitoring time, it means that the processing efficiency of the slave device is abnormal. When the processing efficiency of the slave device is abnormal, the above-mentioned main device executes the replacement strategy, and the main device selects a slave device from the load registration table as a temporary device. The above-mentioned main device sends the first image data to the temporary device again for inference service.

[0034] A further technical solution is that when the master device executes the replacement strategy, the master device downloads the current load registration table from the service again, removes the slave devices marked as working in the load registration table, and re-establishes a new priority table through the load information of the remaining slave devices; the master device selects the front slave device from the new priority table as a temporary device.

[0035] A further technical solution is that when the main device receives the image data returned by the temporary device within the monitoring time, the temporary device replaces the slave device with abnormal efficiency; at the same time, the main device sends an update request to the server, and the server is used to mark the slave device with abnormal processing efficiency as abnormal in the load registration table.

[0036] Preferably, the load information recorded in the load registration table includes the total CPU occupancy rate, the total NPU occupancy rate, the memory occupancy rate and the network delay data; the calculation formula of the load health P in the above step S210 is:

[0037] P=K C ×log(1-C+z)+K N ×log(1-N+z)+K m ×log(1-m+z)+K D ×e -α·D ;

[0038] Where C is the total CPU occupancy, N is the total NPU occupancy, m is the memory occupancy, D is the network delay in milliseconds, and K is the total CPU occupancy. C K is the weight coefficient of the total CPU occupancy in the priority sorting, N K is the weight coefficient of the total NPU occupancy rate in the priority sorting; D K is the weight coefficient of network delay in priority sorting; m is the weight coefficient of memory usage in priority sorting; z is the initial coefficient set between 0 and 0.001; α is a positive proportional coefficient and is used to adjust the impact of network delay. C +K N +K D +K m =1, the above C, N and m all range from 0 to 1, and the above 1 represents 100% occupancy.

[0039] A further technical solution is that it also includes step S500, wherein the server periodically collects the load information of the edge device and updates the load registry based on the acquired load information; the server sets a threshold value of the load health P, and when the load health P of any edge device is lower than the threshold value, the server resets and selects an edge device from the load registry for replacement, and updates the identity in the load registry.

[0040] Compared with the prior art, the beneficial effects of the present invention are at least one of the following:

[0041] The edge device of the present invention adopts a collaborative architecture of a master device and multiple slave devices. In view of the particularity of the video stream, the master device decodes the video stream into a picture format and distributes it to different slave devices. The video stream is decoded into a picture format, which can reduce the amount of data that needs to be transmitted through the network. The master device can monitor the operating status of the slave device in real time. Combined with the load registry established by the server, on the one hand, the server can flexibly adapt to the dynamic joining and exiting of the edge device to ensure the stable connection between the master device and the slave device. On the other hand, the setting of the load registry is conducive to selecting edge devices with lighter loads and stronger processing capabilities. The status of the edge device is obtained through the load health and a priority table is established. The priority table is combined with real-time updated load information, so that the selected device can quickly adapt to changes in the network or hardware status, and the edge computing resources are ensured to be efficiently utilized through a dynamic resource allocation mechanism.

[0042] The present invention performs inference services on the received first image data from the slave device and ensures the high efficiency of data processing and transmission by means of a load balancing interface. In particular, for scenarios with large data traffic or frequent computing requests, the present invention can effectively disperse processing pressure through intelligent load balancing, avoid single-point overload, and speed up data processing and forwarding. At the same time, the processed image data is forwarded to the server via the main device, and then pushed to the front-end web page in real time through the websocket channel, thereby realizing the real-time visualization of the data processing results.

[0043] When faced with high-concurrency tasks, the present invention can effectively reduce the amount of data that needs to be transmitted through the network through selective processing of key frames and intermediate frames, reduce network bandwidth consumption, improve data processing efficiency, and ensure the stability and efficiency of the entire processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. in a certain specific working state. If the specific posture changes, the directional indication will also change accordingly. In the present invention, unless otherwise clearly specified and limited, the term "connection" and the like should be understood in a broad sense. For example, "connection" can be an electrical signal connection or a signal connection; it can also be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] If there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme that satisfies both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0048] Existing edge devices often use Docker container technology to deploy video acquisition services, so that edge devices can isolate and run multiple services on the same device. Its video acquisition service often uses cross-platform audio and video processing tools such as FFmpeg to support more audio and video codecs and conversions. By deploying the video acquisition service on the edge device, the service can collect video streams periodically or in real time.

[0049] refer to Figure 1As shown, one embodiment of the present invention is an architecture method based on a video edge AI inference cluster, comprising the following steps: S100, configuring an inference cluster, the inference cluster comprising multiple edge devices, the edge devices being connected to a server; signal interaction between the edge devices; the server designating an edge device as a master device, and designating multiple edge devices as slave devices. Among them, there are N edge devices, N is a positive integer, and N≥3, wherein the specific value of N can be determined based on available resources and expected load. The server designates an edge device as a master device, and designates N-1 edge devices as slave devices. On the one hand, the architecture of a single master device corresponding to multiple slave devices can achieve a clear division of labor and collaboration mechanism by setting different roles of the master device and the slave device; and provide a basis for the master device to coordinate the slave devices for dynamic task allocation.

[0050] S200, the server configures a load registry, which contains the identity, load information and priority of each edge device; the master device obtains the load registry through the server and establishes an allocation decision. By setting up a load registry to manage the status and load information of all edge devices, the master device can directly grasp the real-time status of the inference cluster through the server, and obtain key information such as the identity, current load and priority of each edge device; when the master device needs to infer from the device, it ensures the performance and identity of each edge device by downloading the load registry, avoiding some edge devices from being overloaded or idle for a long time, and objectively improving the overall utilization of resources.

[0051] For reference, the main device can obtain the load registry in time-sharing. After obtaining the load registry each time, the main device can determine the status of different edge devices. Especially when a failure occurs, the main device needs to respond quickly. The main device can select the edge device through the load registry and quickly transfer the task from the failed edge device to other healthy devices for continued execution. When individual edge devices fail, it is helpful to ensure the continuous operation and high availability of the entire system.

[0052] S300, the main device acquires the video stream of the data acquisition node and decodes and converts the video stream into a picture format to obtain a plurality of continuous first image data; wherein the first image data comprises a plurality of single-frame pictures of consecutive frames. The video stream is a digital signal of the video captured in real time by the data acquisition node; the main device acquires the video stream of the data acquisition node and decodes and converts the video stream into a picture format, and the picture format is more conducive to reasoning than the video format.

[0053] At the same time, the master device accesses the load registry and sends the first image data to the corresponding slave device according to the allocation decision, and distributes the decoded first image data to different slave devices. Each slave device calls the inference model to perform inference services, which realizes the distributed processing of computing tasks, and the slave device calls the inference model to perform inference services.

[0054] For reference, when N=3, a master device and two slave devices are usually used. The master device alternately assigns a number of continuous first image data to two slave devices for reasoning according to the real-time load of the slave devices. By allocating tasks alternately, it is avoided that a slave device is overloaded at a certain moment while the other is idle. In addition, alternating reasoning between the two slave devices can also produce the effect of task parallel processing. Compared with the existing serial processing, this method can significantly increase the speed of system data processing and reduce the total response time of data processing. This method is also conducive to reducing the risk of overload when the slave device is alternating.

[0055] For reference, when N>3, an architecture of 1 master device and multiple slave devices is also adopted. The master device collects and decodes the video stream from the data acquisition node, and then effectively assigns tasks to multiple slave devices based on the real-time load registry. This method allows the master device to have more options when allocating tasks. The master device can selectively send the first image data to the slave device with a lighter load according to the load situation. When a slave device performs inference, the master device transmits other first image data to the slave device with a lighter load in sequence to achieve transmission continuity and further reduce the risk of overload.

[0056] The above-mentioned master device establishing allocation decision includes the following steps:

[0057] S210, the master device downloads the load registration table from the server and extracts the load information, and calculates the load health through the load information. The master device downloads the load registration table from the server, obtains the load information of each slave device, and by calculating the load health, the master device can more accurately evaluate the working status and remaining processing capacity of each slave device, which is conducive to the subsequent selection of slave devices with lighter loads for full utilization.

[0058] S220, sorting the slave devices in the load registration table according to the load health to generate a priority table. The slave devices are sorted according to the calculated load health to obtain a priority table, wherein the priority table is arranged from high to low according to the load health of each slave device. The master device can determine the slave device with high load health through the priority table to facilitate task allocation. The priority table can simplify the decision-making process of the master device.

[0059] S230, the master device selects a slave device from the priority table according to the set number to establish an initial connection and obtain the number of task processes of the slave device; the master device determines the allocation order according to the number of task processes. By establishing the initial connection and obtaining the number of task processes, the master device can further determine whether there is a currently executed process of the slave device. The master device determines the allocation order according to the number of task processes, which is mainly used to determine the task allocation order when allocating, thereby reducing the waiting time of the task in the queue, so that the sent task can be processed in time, and the accuracy and real-time performance of the allocation are improved.

[0060] When the master device and the slave device establish an initial connection, the master device feeds back the connection status to the server, and the server marks the corresponding slave device in the load registration table as working. The feedback connection method allows the status of the slave device to be sent to the server, which is also helpful for monitoring and management of the entire system. Through the working mark, it is conducive to continuously tracking the working status and health of the slave device.

[0061] Based on the above embodiments, another embodiment of the present invention is to further include S400, wherein the slave device generates image data using an inference service for the first image data, and the image data is returned to the master device through a load balancing interface, and the master device forwards the image data to a server, which displays the image data to a front-end web page through a websocket channel.

[0062] For reference, existing edge devices often have limited resources, and the load balancing interface can use the existing Nginx load balancing, HAProxy load balancing and other existing load balancers. In principle, it needs to be suitable for lightweight load balancing.

[0063] By adopting the load balancing interface, the bottleneck that may be caused by a single network path can be avoided, the reliability and stability of the data transmission process can be enhanced, and the data return can be relatively smooth and timely; its server uses the WebSocket channel to achieve two-way communication with the front end, ensuring the real-time and smoothness of data transmission.

[0064] A further technical solution is that when the slave device performs the inference service, the slave device calls the inference model and generates picture data from the first image data through the inference model; the execution steps of the inference service are:

[0065] S410, extract all single-frame pictures in the first image data, and determine the key frames and intermediate frames in the first image data. Among them, the key frame is to obtain the inflection point of the change of the video content, and the intermediate frame represents the content between two adjacent key frames. Extracting the key frames and intermediate frames, under the premise of not directly affecting the accuracy of the video information, selecting the necessary key frames and intermediate frames can reduce the amount of data that needs to be processed. By processing and extracting all single-frame pictures of the first image data, unnecessary processed images are avoided, so that the number of pictures that need to be processed by the slave device is reduced; it is expected that when the device performs data processing in the later stage, the amount of unnecessary image processing is reduced, thereby reducing the computational complexity and processing time of the inference service.

[0066] S420, setting the single-frame image corresponding to the key frame as the key image, and setting the single-frame image adjacent to the key image as the verification image. The key image is used to present key information in the video, and the verification image is used to verify the continuity and accuracy of this information; the key image is used to ensure that key changes in the video stream can be obtained, and the adjacent single-frame images are used to present the transition situation and to check and ensure the consistency and coherence of the key information. There are multiple key frames extracted from the first image data, and each key frame is set as a key image.

[0067] S430, the slave device calls the built-in preset model to perform data processing on the key image and the verification image respectively, and the preset model extracts the edge features of the key image and the verification image to generate a first edge image and a second edge image respectively. This method mainly generates edge images by extracting edge features of the image, which can reduce the dimension of the data, while retaining the structural information of the key image and the verification image, which is beneficial to subsequent data processing, and finds the edge by identifying the places where the brightness changes significantly in the image, which helps to reduce background noise and focus on the structural presentation on the image.

[0068] It should also be noted that the acquisition of the first edge image and the second edge image can also forward data when necessary, thereby serving as a data source for machine learning. The preset model is an existing model and is not specifically limited here.

[0069] For reference, the preset model is an existing model, which can be any of the existing VGGNet, ResNet or Inception models; for example, the existing VGGNet model gradually reduces the spatial dimension of the feature map by stacking multiple layers of small-sized convolution kernels and using the maximum pooling layer, while increasing the depth of the network, so as to perform the feature extraction task in the image faster. The existing ResNet model introduces residual connections, so that the network can easily learn identity mapping, thereby reducing the gradient disappearance problem when extracting features. The existing Inception model adopts a multi-scale processing method, through multiple convolution kernels of different sizes to process the same input in parallel, it can capture various scale information of the image at the same time, can more effectively capture features of different sizes in the image, facilitate the fusion of feature information, and help maintain the rationality of the edge image. In theory, the existing model can extract image edge features and generate new images.

[0070] S440, the above-mentioned slave device sets a similarity parameter, and judges the similarity between the first edge image of the key image and the second edge image of the corresponding verification image to determine whether the selection of the key frame is normal; when the similarity is greater than or equal to the similarity parameter, it indicates that the selection of the key frame is normal, and the above-mentioned slave device compresses the first edge image to obtain a compressed image, and executes S450; when the similarity is less than the similarity parameter, it indicates that the selection of the key frame is abnormal, and the key frame is adjusted to an intermediate frame.

[0071] Among them, the preset similarity parameter is a set threshold. The similarity parameter is set to compare the first edge image with the second edge image to determine the degree of difference between two adjacent frames in the video stream. The similarity parameter mainly determines whether the difference between adjacent images is within a controllable range. If it is within the controllable range, it means that the recognition of key change points is met.

[0072] Specifically, if the similarity between the two edge images is greater than or equal to the set similarity parameter, the selection of the current key frame is considered normal, which means that the difference between the key frame and its adjacent frames is within an acceptable range and can effectively represent a key change point in the video stream. For example, if two consecutive frames are the same, it means continuous static, which usually indicates a key change point for the entire action. On the contrary, if the similarity is less than the set parameter, it means that the selected key frame has changed too much from the adjacent content, and video frame drops may occur or the sampling segment may be maliciously modified. If its data is used in the later stage, it may have a high deviation, so it is adjusted to an intermediate frame.

[0073] S450: Integrate the compressed pictures into picture data, where the picture data includes all compressed pictures corresponding to the first image data.

[0074] Furthermore, in order to maintain the edge device running under an ideal load as much as possible and reduce the fluctuation of the slave device load, when executing S410, the above-mentioned slave device sets the picture accumulation number; obtains the number of all single-frame pictures in the first image data and determines whether the number of single-frame pictures exceeds the picture accumulation number; wherein the picture accumulation number is the picture processing amount of the word under the relatively ideal state of the slave device.

[0075] When the number of single-frame images exceeds the cumulative number, it is generally believed that the allocated execution data volume is relatively large. Although it can also be overloaded for processing, the high-intensity overload situation may affect the life of the slave device, and when overloaded, there is a risk of untimely response speed during the reasoning process. In order to reduce unnecessary risks, the above-mentioned slave device executes a preset dynamic strategy, and the slave device divides the first image data into second image data and third image data, and the above-mentioned second image data is used to execute steps S420 to S450. Among them, the second image data contains the same number of single-frame images as the cumulative number of images. The processing data of the first image data is divided into two parts by segmentation means, and the single-frame image of the second image data is the same as the cumulative number of images, so that when the slave device directly processes the second image data, the amount of single-frame image data processed by it will not exceed the single-frame image processing amount of the slave device in a relatively ideal state.

[0076] The third image data includes single-frame pictures whose first image data exceeds the cumulative number. The number of single-frame pictures of the third image data is the number of single-frame pictures remaining after subtracting the second image data from the first image data.

[0077] When the number of single-frame images exceeds the cumulative number, in order to coordinate processing, the above-mentioned slave device sends a request to the server, the server selects an idle edge device from the load registry and sets the identity to the assisting device, and the slave device forwards the third image data to the assisting device, and the assisting device executes steps S420 to S450.

[0078] It should be noted that if the total number of single-frame images of the first image data is more than twice the total number of single-frame images of the second image data, when the third image data executes steps S420 to S450 on the assisting device, the third image data also needs to be segmented in the same manner as the first image data, and the assisting device will also send a request to the server, which will select an idle edge device for assistance.

[0079] Furthermore, when executing S440, the similarity parameter set by the slave device is T, and the similarity between the first edge image and the corresponding second edge image is obtained; the calculation formula of the similarity S is:

[0080]

[0081] In the formula, u A is the average brightness of the first edge image; u B is the average brightness of the second edge image; β AB is the covariance between the first edge image and the second edge image; is the variance of the first edge image; is the variance of the second edge picture; L1 and L2 are the stability constants of the first edge picture and the second edge picture respectively.

[0082] When S≥T, the selection of the key frame is considered normal; when S<T, the selection of the key frame is considered abnormal.

[0083] For reference, the acquisition of average brightness usually requires adjusting the first edge image and the second edge image to the same grayscale; the brightness value is processed according to the basic processing specifications with black to white and a brightness parameter of 0 to 255, and the sum of the brightness of the image pixels and the total number of pixels are obtained at the same grayscale to obtain the average brightness.

[0084] For reference, we traverse the pixels of the two images to obtain the brightness values, subtract the average brightness value from each pixel brightness value, and then take the average after multiplying them to get the covariance. AB The calculation formula can be:

[0085]

[0086] In the formula, A i is the brightness value of the pixel at the i-th point in the first edge image, B i is the brightness value of the i-th pixel in the second edge image, and its u A and u B are the average brightness of the first edge picture and the second edge picture respectively, and n is the total number of pixels of the first edge picture and the second edge picture.

[0087] By setting a stability constant, the stability constant is usually less than 0.001, and the setting of the stability constant is mainly used to adjust the quality difference impact according to the noise level of the image stream.

[0088] One implementation manner is that when the number of second edge pictures is equal to 1, the first edge picture is a start frame or an end frame of the first image data, and the similarity S between the first edge picture and the corresponding second edge picture is directly obtained by the above.

[0089] One implementation method is that when the number of second edge pictures is equal to 2, the second edge pictures are respectively the two frames before and after the first edge picture; the familiarity S1 between the second edge picture of the previous frame and the first edge picture, and the familiarity S2 between the second edge picture of the next frame and the first edge picture are obtained according to the above formula; the similarity S between the first edge picture and the corresponding second edge picture is the average value of S1 and S2.

[0090] Based on the above embodiments, another embodiment of the present invention is that in S300, when the master device sends the first image data to the slave device for reasoning, the master device sets a monitoring time, and the master device continuously monitors the progress of the reasoning of the corresponding slave device during the monitoring time, and determines whether the processing efficiency of the slave device is normal.

[0091] For reference, when the master device sends the first image data to the slave device for inference processing, the master device needs to set a monitoring time. The monitoring time is usually a time period, during which the master device monitors the work progress of each slave device in real time. The master device's monitoring is mainly to obtain the task progress of the slave device, and determine the progress information through the task progress. The progress information includes the processing status of the slave device, the amount of work completed, and the error report within the monitoring time.

[0092] The master device uses the monitoring time to determine whether the processing efficiency of the slave device is normal. This determination is based on whether the inference result, i.e., the image data, returned by the slave device is received within the set monitoring time.

[0093] When the master device receives the image data returned by the slave device within the monitoring time, it indicates that the slave device can complete the assigned reasoning task within a reasonable time, which means that the processing efficiency of the slave device is normal.

[0094] When the master device does not receive the image data within the monitoring time, it indicates that the processing efficiency of the slave device is abnormal. The abnormal processing efficiency of the slave device usually means that the slave device cannot complete the task within the expected time, which may be caused by insufficient processing capacity, system failure or service interruption.

[0095] When the processing efficiency of the slave device is abnormal, the master device adopts a replacement strategy to maintain the overall operating efficiency and stability of the system; at this time, the master device executes the replacement strategy, and the master device re-accesses or downloads the updated load registry to obtain the load information and status of all slave devices; the master device selects a slave device from the load registry as a temporary device. Then the master device sends the first image data to the temporary device again for inference service. By reallocating tasks to other available slave devices, it can be ensured that the task chain is not interrupted.

[0096] Furthermore, in order to better optimize resource allocation and increase robustness, when the master device executes the replacement strategy, the master device downloads the current load registry from the service again, removes the slave devices marked as working in the load registry, and re-establishes a new priority table through the load information of the remaining slave devices; the master device selects the front slave device from the new priority table as a temporary device.

[0097] On the one hand, slave devices with working marks have usually been assigned tasks. Reassigning new tasks to these marked slave devices may cause task overlap and overload of processing capacity, thus affecting the overall performance of the system and the completion time of the task. Therefore, downloading the load registry to promptly remove the slave devices marked with working marks is to avoid accidental assignment of tasks to slave devices that already have tasks but are not immediately inferred due to task intervals, ensuring that new tasks can be assigned to devices that are currently idle or lightly loaded, thereby improving the efficiency of task processing and balancing the system load. In the event of abnormal processing efficiency or failure of a slave device, the load registry is dynamically updated and the load status and priority of the slave device are re-evaluated, and a response is made to the abnormal status of the slave device, thereby effectively reducing the impact of the failure or excessive load of a single slave device on the performance of the entire system.

[0098] For reference, when the entire inference cluster is in a high-concurrency environment, it is crucial to maintain a balanced distribution of tasks between the master and slaves. If tasks are constantly assigned to busy slaves, it is easy to cause the processing queue to become longer, and the response time and processing time delay to increase. When the master executes the replacement strategy, it will remove busy slaves as much as possible, so that the master can assign tasks to slaves that are currently more capable of processing new tasks, which can also reduce the risk of processing time delays caused by high concurrency.

[0099] Furthermore, in order to improve task continuity and response speed, the above-mentioned master device receives the image data returned by the temporary device within the monitoring time, and the temporary device replaces the slave device with abnormal efficiency; because this situation shows that the temporary device meets the requirements in the post-processing data speed, even if the problem of the slave device with abnormal efficiency is a temporary problem (network problem causes performance down), but in this case, in order to avoid uncertain factors, directly use the temporary device to replace the slave device with abnormal efficiency to work, which can achieve the ability of rapid recovery and improve the ability of the inference cluster to resist external interference.

[0100] At the same time, the master device sends an update request to the server, and the server is used to mark the slave device with abnormal processing efficiency as abnormal in the load registration table. By sending an update request to the server and marking the slave device with abnormal efficiency as abnormal in the load registration table, it is beneficial to the later fault diagnosis.

[0101] Based on the above embodiment, another embodiment of the present invention is that the load information recorded in the above load registration table includes the total CPU occupancy rate, the total NPU occupancy rate, the memory occupancy rate and the network delay data; the calculation formula of the load health P in the above step S210 is:

[0102] P=K C ×log(1-C+z)+K N ×log(1-N+z)+K m ×log(1-m+z)+K D ×e -α·D ;

[0103] Where C is the total CPU occupancy, N is the total NPU occupancy, m is the memory occupancy, D is the network delay in milliseconds, and K is the total CPU occupancy. C K is the weight coefficient of the total CPU occupancy in the priority sorting, N K is the weight coefficient of the total NPU occupancy rate in the priority sorting; D K is the weight coefficient of network delay in priority sorting; m is the weight coefficient of memory usage in priority sorting; z is the initial coefficient set between 0 and 0.001; α is a positive proportional coefficient and is used to adjust the impact of network delay.

[0104] Among them, the value range of C, N and m in the formula is 0 to 1, and 1 means 100% occupancy. Similarly, the idle rate of CPU, idle rate of NPU and idle rate of memory are obtained through 1-C, 1-N and 1-m in the formula respectively. The idle rate of CPU is an important indicator to measure the overall load of the system and the utilization of computing resources. The idle rate of NPU is to measure the processing capacity of the task and determine the executable resources. The idle rate of memory is mainly used to determine the basic resources for running any computing task to reduce the risk of memory overflow that may cause system crash.

[0105] The score is quantified by using a nonlinear function, with z as the initial coefficient, and its value can be very small, such as 0.0001, mainly used to avoid the situation where the function calculates a score of zero.

[0106] Among them, network delay is the influencing data, and it is necessary to convert network delay into a score to make the score type consistent. Therefore, e -α·D The negative impact of delay on the score is quantified by using a decreasing nonlinear function. At the same time, the rating score of network delay can be adjusted by using a positive proportional coefficient α.

[0107] Among them, K C +K N +K D +K m=1, mainly used to indicate the priority weight of different load information.

[0108] Based on the above embodiments, another embodiment of the present invention is, in order to form reasonable task allocation decisions during the continuous operation of the inference cluster, it also includes step S500, the above server regularly collects the load information of the edge device and updates the load registry according to the acquired load information; the above server will actively request the load information of all edge devices within a set time interval, which is used to evaluate the current load status of each edge device, and organize and update the collected load information to the load registry, so as to achieve the purpose of real-time update of the load registry, so that the main device is conducive to grasping the operating status of the edge device at any time, so as to facilitate dynamic task allocation.

[0109] The server sets a threshold value of the load health P. When the load health P of any edge device is lower than the threshold value, the server selects an edge device from the load registry for replacement and updates the identity in the load registry.

[0110] For reference, the server will set the corresponding load health P threshold according to historical data or system needs. The threshold represents the lowest limit for the edge device to perform tasks normally. When the load health P set by the edge device is above the threshold, it usually means that the edge device can perform tasks smoothly. When the load health P of the edge device is lower than the threshold, it is generally believed that the edge device is not competent for the task, and it may be affected after participating in the task, and there may be a high risk of processing delay, task failure or forced restart.

[0111] When working, if the load health of an edge device in the inference cluster is below the threshold for a long time, the server will use the information in the load registry to select a spare edge device for replacement. Since the newly selected device usually has a lower load and better health status, it can quickly take over the tasks of the device that is about to fail, and after replacing the new device to a working state, the server will update the load registry to reflect the task architecture of the current inference cluster to determine that the edge device is marked as available, and mark the status of the inefficient edge device, so that the main device will not assign tasks to the marked inefficient edge device again in subsequent task allocation.

[0112] The "one embodiment", "another embodiment", "embodiment", "preferred embodiment" and the like mentioned in this specification refer to the specific features, structures or characteristics described in conjunction with the embodiment included in at least one embodiment generally described in this application. The same expression appearing in multiple places in the specification does not necessarily refer to the same embodiment. Further, when a specific feature, structure or characteristic is described in conjunction with any embodiment, it is claimed that the realization of such feature, structure or characteristic in conjunction with other embodiments also falls within the scope of the present invention.

[0113] Although the present invention is described herein with reference to a number of illustrative embodiments of the present invention, it will be appreciated that those skilled in the art may devise many other modifications and implementations that fall within the scope and spirit of the principles disclosed herein. More specifically, within the scope of the present disclosure, drawings, and claims, a variety of variations and modifications may be made to the components and / or layout of the subject combination layout. In addition to the variations and modifications made to the components and / or layout, other uses will also be apparent to those skilled in the art.

Claims

1. An architecture method based on video edge AI inference cluster, characterized in that: The steps include: S100, configuring an inference cluster, the inference cluster including multiple edge devices, the edge devices accessing a server; signal exchange between the edge devices; the server designating an edge device as a master device and designating multiple edge devices as slave devices; S200, the server configures a load registry, the load registry includes the identity, load information and priority of each edge device; the master device obtains the load registry through the server and establishes an allocation decision; S300, the master device acquires the video stream of the data acquisition node and decodes the video stream into a picture format to obtain a plurality of continuous first image data; at the same time, the master device accesses the load registry and sends the first image data to the corresponding slave device according to the allocation decision, and the slave device calls the inference model to perform inference service; The video stream is a digital signal of a video captured in real time by a data acquisition node; the first image data includes a single-frame image of several consecutive frames; The master device establishes the allocation decision including the following steps: S210, the master device downloads the load registration table from the server and extracts the load information, and calculates the load health through the load information. S220, sorting the slave devices in the load registration table according to the load health to generate a priority table; S230, the master device selects a slave device from the priority table according to the set number to establish an initial connection and obtains the number of task processes of the slave device; the master device determines the allocation order according to the number of task processes; When the master device establishes an initial connection with the slave device, the master device feeds back the connection status to the server, and the server marks the corresponding slave device in the load registration table as working.

2. The architecture method based on the video edge AI inference cluster according to claim 1 is characterized in that: It also includes S400, the slave device generates picture data from the first image data through an inference service, the picture data is returned to the master device through a load balancing interface, the master device forwards the picture data to the server, and the server displays it to the front-end web page through a channel websocket.

3. The architecture method based on the video edge AI inference cluster according to claim 2, characterized in that: When the slave device performs the inference service, the slave device calls the inference model and generates picture data from the first image data through the inference model; the execution steps of the inference service are: S410, extracting all single-frame pictures in the first image data, and determining key frames and intermediate frames in the first image data; S420, setting the single-frame picture corresponding to the key frame as the key picture, and setting the single-frame picture adjacent to the key picture as the verification picture; There are multiple key frames extracted from the first image data, and each key frame is set as a key picture; S430, calling a built-in preset model from the device to perform data processing on the key image and the verification image respectively, and extracting edge features of the key image and the verification image by the preset model to generate a first edge image and a second edge image respectively; S440, the slave device sets a similarity parameter, performs similarity judgment on a first edge picture of the key picture and a second edge picture of the corresponding verification picture, and determines whether the selection of the key frame is normal; When the similarity is greater than or equal to the similarity parameter, it indicates that the selection of the key frame is normal, the slave device compresses the first edge image to obtain a compressed image, and executes S450; When the similarity is less than the similarity parameter, it means that the key frame selection is abnormal, and the key frame is adjusted to an intermediate frame; S450: Integrate the compressed pictures into picture data, where the picture data includes all compressed pictures corresponding to the first image data.

4. The architecture method based on the video edge AI inference cluster according to claim 3 is characterized in that: When executing S410, the slave device sets a cumulative number; obtains the number of all single-frame pictures in the first image data and determines whether the number of single-frame pictures exceeds the cumulative number of pictures; When the number of single-frame images exceeds the cumulative number, the slave device executes a preset dynamic strategy, and the slave device divides the first image data into second image data and third image data, and the second image data is used to execute steps S420 to S450; The second image data includes single-frame images of the same number as the accumulated number of images; The third image data includes a single frame of the first image data exceeding the cumulative number; The slave device sends a request to the server, the server selects an idle edge device from the load registry and sets the identity to the assisting device, the slave device forwards the third image data to the assisting device, and the assisting device executes steps S420 to S450.

5. The architecture method based on video edge AI inference cluster according to claim 3 is characterized in that: When executing S440, the similarity parameter set by the slave device is T, and the similarity between the first edge picture and the corresponding second edge picture is obtained; the calculation formula of the similarity S is: In the formula, u A is the average brightness of the first edge image; u B is the average brightness of the second edge image; β AB is the covariance between the first edge image and the second edge image; is the variance of the first edge image; is the variance of the second edge image; L1 and L2 are the stability constants of the first edge image and the second edge image respectively; When S≥T, it is considered that the selection of the key frame is normal; when S<T, it is considered that the selection of the key frame is abnormal.

6. The architecture method based on video edge AI inference cluster according to claim 1, characterized in that: In S300, when the master device sends the first image data to the slave device for inference, the master device sets a monitoring time, during which the master device continuously monitors the progress of the inference of the corresponding slave device and determines whether the processing efficiency of the slave device is normal; If the master device receives the image data returned by the slave device within the monitoring time, it means that the processing efficiency of the slave device is normal; Among them, if the main device does not receive the image data within the monitoring time, it means that the processing efficiency of the slave device is abnormal. When the processing efficiency of the slave device is abnormal, the main device executes the replacement strategy, and the main device selects a slave device from the load registration table as a temporary device. The main device sends the first image data to the temporary device again for inference service.

7. The architecture method based on video edge AI reasoning cluster according to claim 6, characterized in that: When the master device executes the replacement strategy, the master device downloads the current load registration table from the service again, removes the slave devices marked as working in the load registration table, and re-establishes a new priority table through the load information of the remaining slave devices; the master device selects the front slave device from the new priority table as a temporary device.

8. The architecture method based on video edge AI inference cluster according to claim 6, characterized in that: When the master device receives the image data returned by the temporary device within the monitoring time, the temporary device replaces the slave device with abnormal efficiency; at the same time, the master device sends an update request to the server, and the server is used to mark the slave device with abnormal processing efficiency as abnormal in the load registration table.

9. The architecture method based on video edge AI inference cluster according to claim 1, characterized in that: The load information recorded in the load registration table includes the total CPU occupancy rate, the total NPU occupancy rate, the memory occupancy rate and the network delay data; the calculation formula of the load health P in step S210 is: P=K C ×log(1-C+z)+K N ×log(1-N+z)+K m ×log(1-m+z)+K D ×e -α·D ; Where C is the total CPU occupancy, N is the total NPU occupancy, m is the memory occupancy, D is the network delay in milliseconds, and K is the total CPU occupancy. C K is the weight coefficient of the total CPU occupancy in the priority sorting, N K is the weight coefficient of the total NPU occupancy rate in the priority sorting; D K is the weight coefficient of network delay in priority sorting; m is the weight coefficient of memory usage in priority sorting; z is the initial coefficient set between 0 and 0.001; α is a positive proportional coefficient and is used to adjust the impact of network delay.

10. The architecture method based on video edge AI inference cluster according to claim 1, characterized in that: It also includes step S500, wherein the server periodically collects load information of the edge device and updates the load registry according to the acquired load information; the server sets a threshold value of the load health P, and when the load health P of any edge device is lower than the threshold value, the server resets and selects an edge device from the load registry for replacement, and updates the identity in the load registry.

Images