Converged communication video stream processing method and device based on edge calculation

Abstract

The invention provides a converged communication video stream processing method and device based on edge calculation, and the method comprises the steps: carrying out the scene switching detection of each frame of image in a video stream, and if the scene switching is detected, extracting the image, and determining the image as a candidate image. If scene switching is not detected at a preset time interval, importance evaluation is carried out on the image, and if an evaluation score is greater than a first preset threshold value and the number of image frames extracted in a time period corresponding to the preset time interval does not reach a second preset threshold value, the image is extracted and determined as a candidate image. After scene switching detection of each frame of image is completed, the optimal frame of image is selected from the candidate images and sent to the server to determine feedback information, so that the efficiency and accuracy of video stream processing can be improved. And the processing process is completed through the converged communication platform, and the terminal equipment only needs to upload the video stream to the converged communication platform, so that the efficiency and the quality of video stream processing can be improved while the intelligent cost is maintained.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for processing fused communication video streams based on edge computing. Background Art

[0002] With the rapid development of information technology, IoT devices have higher and higher requirements for real-time data processing. In order to reduce latency, edge computing can be used to push part of the data processing to the edge of the network, so that it can be close to the data source or user end, and data can be processed on the spot, thereby effectively reducing data transmission delays and reducing network bandwidth load. For example, in an intelligent monitoring system, some smart cameras can perform some simple video stream analysis locally, and the results of these local processing will be transmitted to the server for further analysis and storage.

[0003] However, in the process of further pursuing the efficiency of video stream processing, in order to improve the efficiency of video stream processing, a large number of ordinary terminals may need to be replaced with intelligent terminals, and the computing and processing capabilities of intelligent terminals need to be improved, which will greatly increase the cost of intelligence. In summary, it is difficult for existing video stream processing methods to improve the efficiency and quality of video stream processing while maintaining a certain intelligent cost. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defect that the video stream processing method in the prior art is difficult to improve the efficiency and quality of video stream processing while maintaining a certain intelligent cost.

[0005] In a first aspect, the present application provides a method for processing a converged communication video stream based on edge computing, which is applied to a converged communication platform, and the method includes:

[0006] Get the video stream in unified encoding format;

[0007] For each frame of the image in the video stream, scene switching detection is performed on the image in turn, and if a scene switching is detected, the image is extracted from the video stream and determined as a candidate image;

[0008] If no scene switching is detected within a preset time interval, an importance evaluation is performed on the image to obtain an evaluation score for the image; if the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, the image is extracted from the video stream and determined as a candidate image;

[0009] After completing the scene switching detection for each frame image in the video stream, selecting the best frame image from each candidate image, and sending the best frame image to the server;

[0010] Determine feedback information determined by the server according to the optimal frame image, and send the feedback information to the user end.

[0011] In one embodiment, performing scene switching detection on the image includes:

[0012] Determining whether the image is the first frame image in the video stream, and if so, confirming that no scene switching is detected for the image;

[0013] If not, determining the previous frame image corresponding to the image in the video stream as the reference image;

[0014] Performing multi-dimensional feature extraction on the image and the reference image respectively to obtain a first feature set corresponding to the image and a second feature set corresponding to the reference image;

[0015] Calculating the difference between the first feature set and the second feature set in each dimension respectively to obtain the difference between the first feature set and the second feature set in each dimension;

[0016] Obtaining the weight of each dimension, and performing weighted summation on the difference between the first feature set and the second feature set in each dimension according to the weight of each dimension to obtain a target difference value;

[0017] If the target difference value is greater than a preset threshold, it is confirmed that a scene switch is detected for the image;

[0018] If the target difference value is not greater than the preset threshold, it is confirmed that no scene switching is detected for the image.

[0019] In one embodiment, the step of performing importance assessment on the image to obtain an assessment score of the image includes:

[0020] Get the preset target detection model;

[0021] Performing target detection on the image using the target detection model to obtain a target detection result;

[0022] Based on a preset importance assignment table, the target detection result is evaluated for importance to obtain an importance score of the target detection result on the importance assignment table, and the importance score is determined as the evaluation score of the image.

[0023] In one embodiment, selecting the best frame image from each candidate image and sending the best frame image to the server includes:

[0024] For each candidate image, calculating its edge sharpness based on the candidate image, and determining the clarity of the candidate image according to the edge sharpness;

[0025] Determine the corresponding information entropy according to the grayscale of the candidate image, and measure the correlation between the candidate image and the preset target feature to obtain a correlation quantization value;

[0026] Based on the clarity, information entropy and correlation quantization value of the candidate image, the candidate image is scored to obtain a quality score of the candidate image;

[0027] When the quality score of each candidate image is determined, the candidate image with the highest quality score is selected as the optimal frame image, and the optimal frame image is sent to the server.

[0028] In one embodiment, scoring the candidate image based on the clarity, information entropy, and correlation quantization value of the candidate image to obtain the quality score of the candidate image includes:

[0029] Get the task requirements for this video stream processing;

[0030] Determine the clarity, information entropy and weights of the correlation quantization value of the candidate image respectively according to the task requirements;

[0031] According to the weights of the clarity, information entropy and correlation quantization value of the candidate image, the clarity, information entropy and correlation quantization value of the candidate image are weighted and summed to obtain the quality score of the candidate image.

[0032] In one embodiment, the determining feedback information determined by the server according to the optimal frame image, and sending the feedback information to the user terminal, includes:

[0033] When the optimal frame image is sent to the server, the server is triggered to match the optimal frame image with a preset black and white list, and after obtaining a matching result, a structured analysis is performed on the optimal frame image to obtain an analysis result, and feedback information is generated according to the matching result and the analysis result and sent to the converged communication platform;

[0034] When the feedback information sent by the server is received, the received feedback information is determined as feedback information determined by the server according to the optimal frame image, and the feedback information is sent to each user terminal corresponding to the preset user list.

[0035] In one embodiment, before sending the optimal frame image to the server, the method further includes:

[0036] The optimal frame image is subjected to noise reduction and size adjustment, and the optimal frame image after noise reduction and size adjustment is encrypted to obtain the encrypted optimal frame image and use it as the optimal frame image finally sent to the server.

[0037] In a second aspect, the present application provides a converged communication video stream processing device based on edge computing, which is applied to a converged communication platform, and the device includes:

[0038] A video stream acquisition module is used to acquire video streams in a unified encoding format;

[0039] A first frame extraction module is used to perform scene switching detection on each frame of the image in the video stream in turn, and if a scene switching is detected, extract the image from the video stream and determine it as a candidate image;

[0040] A second frame extraction module is used to evaluate the importance of the image if no scene switching is detected within a preset time interval to obtain an evaluation score for the image; if the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, extract the image from the video stream and determine it as a candidate image;

[0041] An image selection module, configured to select an optimal frame image from each candidate image after completing scene switching detection for each frame image in the video stream, and send the optimal frame image to a server;

[0042] The information determination module is used to determine the feedback information determined by the server according to the optimal frame image, and send the feedback information to the user end.

[0043] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the edge computing-based converged communication video stream processing method as described in any of the above embodiments.

[0044] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0045] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, they perform the steps of the edge computing-based converged communication video stream processing method as described in any of the above embodiments.

[0046] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0047] The present application provides a method and device for processing a converged communication video stream based on edge computing. The method is applied to a converged communication platform, including: obtaining a video stream in a unified coding format, and performing scene switching detection on each frame of the video stream in turn. If a scene switching is detected, the image is extracted from the video stream and determined as a candidate image. If the scene switching is not detected within a preset time interval, the image is evaluated for importance to obtain an evaluation score for the image. If the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, the image is extracted from the video stream and determined as a candidate image. In this way, on the one hand, a candidate image can be preferentially selected by scene switching, and on the other hand, a candidate image can be selected according to the importance of the image when the scene switching is not detected within a preset time interval, thereby comprehensively determining multiple candidate images based on scene changes and image importance to ensure the quality and information richness of the candidate images. After completing the scene switching detection of each frame of the video stream, the optimal frame image is selected from each candidate image and sent to the server, and then the feedback information determined by the server based on the optimal frame image is determined. By further screening each candidate image to determine the optimal frame image, a more representative and high-quality frame image can be extracted for server-side analysis and processing, which can improve the efficiency of the overall video stream analysis and processing while also improving the accuracy of the video stream analysis and processing, and ensuring the processing quality of the video stream. In addition, the above processing process is completed through the converged communication platform, and the terminal device only needs to upload the video stream to the converged communication platform, which can improve the efficiency and quality of video stream processing while maintaining a certain intelligent cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 A flowchart of a method for processing a converged communication video stream based on edge computing provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a process for performing scene switching detection on an image provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a process for evaluating the importance of an image provided in an embodiment of the present application;

[0052] Figure 4A schematic diagram of a process for selecting an optimal frame image from various candidate images and sending the optimal frame image to a server provided in an embodiment of the present application;

[0053] Figure 5 A scenario application example diagram of a method for processing a converged communication video stream based on edge computing provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of a fusion communication video stream processing device based on edge computing provided in an embodiment of the present application;

[0055] Figure 7 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] In one of the embodiments, the present application provides a method for processing a converged communication video stream based on edge computing, and the following embodiments are described by applying the method to a converged communication platform. It can be understood that the server that executes the converged communication platform method can be a server that deploys the converged communication platform, and the present application does not impose specific restrictions on this. In addition, the subjects involved in the present application include terminal devices, converged communication platforms, and core servers, and the terminal devices, converged communication platforms, and core servers can communicate through the network. Among them, the converged communication platform may refer to an edge device with certain computing and storage capabilities. And the converged communication platform can communicate with the user terminal in a variety of communication methods, such as voice, video calls, etc. In the following embodiments, the core server is referred to as the server.

[0058] like Figure 1 As shown, the present application provides a converged communication video stream processing method based on edge computing, which is applied to a converged communication platform, and the method includes:

[0059] S101: Obtain a video stream in a unified encoding format.

[0060] Specifically, the terminal device can push the simply processed video stream to the converged communication platform. Since the video stream format pushed by the terminal device to the converged communication platform may have different formats, such as RTMP, RTSP, GB / T28181, etc. When the converged communication platform receives the video stream pushed by the terminal device, the converged communication platform performs format detection and conversion on the received video stream, converts all received video streams into a unified encoding format, and obtains the video stream in the unified encoding format, so as to process the video stream later.

[0061] Among them, terminal devices refer to devices with video stream recording functions, such as cameras, drones, etc.

[0062] In one example, the converged communication device may convert the encoding formats of all received video streams into the H.264 encoding format.

[0063] S102: Perform scene switching detection on each frame of the image in the video stream in turn.

[0064] In this step, after obtaining the video stream in a unified encoding format, the corresponding frame images are determined based on the video stream. Each frame image in the video stream is sequentially processed in order of frame number, and scene switching detection is performed on the image to detect whether the scene in the video stream has changed, such as the appearance of targets such as faces or license plates.

[0065] S103: If a scene switch is detected, extract the image from the video stream and determine it as a candidate image.

[0066] In this step, when the result of the scene switching detection of the image is that a scene switching is detected, it means that the scene in the video stream has changed. Since the image when the scene changes is likely to contain important information, the scene switching will be detected. At this time, the converged communication platform can extract the image from the video stream and determine the extracted image as a candidate image.

[0067] It can be understood that the candidate image refers to an image in the video stream that contains more important information or has higher image content importance.

[0068] S104: If no scene switch is detected, determine whether the time interval between the last scene switch detection reaches a preset time length.

[0069] The preset duration may be set or adjusted by comprehensively considering factors such as the length of the video stream, the accuracy of detection, and platform performance. In one example, the preset duration may be set to 10 seconds.

[0070] In this step, when the result of the scene switch detection for the image is that no scene switch is detected, it means that the scene in the video stream has not changed. At this time, the fusion communication platform will determine whether the duration of the last scene switch detected in the current interval reaches the preset duration, so as to determine whether to evaluate the importance of the image.

[0071] S105: If the preset time length is not reached, the image is not determined as a candidate image.

[0072] S106: If no scene switching is detected within a preset time interval, an importance evaluation is performed on the image to obtain an evaluation score for the image.

[0073] Among them, the scoring score is used to measure the matching degree between the information contained in the image and the detection target.

[0074] If the preset duration is not reached, the video stream will not be framed, that is, the image will not be extracted from the video stream and determined as a candidate image. When the preset duration is reached, that is, no scene switching is detected within the preset duration, the fusion communication platform can evaluate the importance of the image to measure the matching degree between the information contained in the image and the detection target, and then obtain the evaluation score of the image.

[0075] Specifically, the detection target refers to the target to be detected during the video stream processing process. For example, when the video stream processing method of the present application is applied to an access control system, the detection target can be set to recognize a face or a license plate.

[0076] It is understandable that when no scene switch is detected within a preset time interval, in order to avoid the final number of extracted and determined candidate images being too small to affect the reliability of subsequent optimal frame image selection, whether to extract and determine the candidate images will be further determined based on importance evaluation.

[0077] S107: Determine whether the evaluation score is greater than a first preset threshold and whether the number of image frames extracted within a time period corresponding to the preset time length does not reach a second preset threshold.

[0078] The first preset threshold and the second preset threshold are both empirical values ​​and can be adjusted according to relevant factors.

[0079] S105: If the evaluation score is not greater than the first preset threshold or the number of image frames extracted within the time period corresponding to the preset time length reaches the second preset threshold, the image is not determined as a candidate image.

[0080] S108: If the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset duration does not reach a second preset threshold, the image is extracted from the video stream and determined as a candidate image.

[0081] The time period corresponding to the preset duration refers to the time period corresponding to the preset duration of the last detected scene switch interval. For example, assuming that the preset duration is 10s, when the last time a scene switch was detected is 9:00:00, if no scene switch is detected within the time interval from 9:00:00 to 9:00:10, then the time period corresponding to the preset duration can be determined as 9:00:00 to 9:00:10. It should be noted that even if the preset duration is the same, if the time when the scene switch was last detected is different, the time period corresponding to the preset duration will also be different.

[0082] When it is determined that it is necessary to further determine whether to extract and determine the candidate image based on the importance evaluation, in order to avoid extracting too many images in the same time interval, the fusion communication platform will determine whether to extract and determine the candidate image based on the evaluation score and the number of image frames extracted in the time period corresponding to the preset duration. If the evaluation score is greater than the first preset threshold, it means that the image contains the detection target. On this basis, if the number of image frames extracted in the time period corresponding to the preset duration does not reach the second preset threshold, the image can be extracted from the video stream and the extracted image can be determined as a candidate image. On the contrary, that is, the situation described in S105, specifically includes the following situations:

[0083] (1) The evaluation score is not greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset duration does not reach a second preset threshold.

[0084] (2) The evaluation score is greater than the first preset threshold and the number of image frames extracted within the time period corresponding to the preset time length reaches the second preset threshold.

[0085] (3) The evaluation score is not greater than the first preset threshold and the number of image frames extracted within the time period corresponding to the preset time length reaches the second preset threshold.

[0086] If any of the above three situations occurs, there is no need to extract and determine candidate images from the video stream.

[0087] S109: After completing the scene switching detection for each frame image in the video stream, an optimal frame image is selected from each candidate image, and the optimal frame image is sent to the server.

[0088] The optimal frame image refers to an image with the best comprehensive quality among all candidate images.

[0089] In this step, after the converged communication platform completes the scene switching detection for each frame image in the video stream, the optimal frame image can be selected from the determined candidate images and the selected optimal frame image can be sent to the server so that the server can perform analysis based on the optimal frame image.

[0090] S110: Determine feedback information determined by the server according to the optimal frame image, and send the feedback information to the user end.

[0091] Specifically, after the server analyzes the optimal frame image, it can obtain corresponding feedback information, and the server can send the feedback information to the converged communication platform. In this way, the converged communication platform can determine the feedback information determined by the server according to the optimal frame image, and then notify the user end of the feedback information.

[0092] Furthermore, in the process of sending feedback information to the user end, the converged communication platform can be linked with the corresponding terminal device of the user end (such as mobile phone, computer, tablet, etc.) to notify the user end of the feedback information in the form of voice, text or video call.

[0093] It is understandable that since the converged communication platform is also an edge device, using the converged communication platform to evaluate and screen images in the video stream can reduce the pressure on the server while reducing the delay of large-scale data transmission. It also avoids the situation where ordinary terminal devices have to be replaced on a large scale due to performance requirements. At the same time, the computing performance requirements of the terminal devices are also relatively low. This improves the efficiency and quality of video stream processing while maintaining a certain cost.

[0094] The present application provides a method and device for processing a converged communication video stream based on edge computing. The method is applied to a converged communication platform, including: obtaining a video stream in a unified coding format, and performing scene switching detection on each frame of the video stream in turn. If a scene switching is detected, the image is extracted from the video stream and determined as a candidate image. If the scene switching is not detected within a preset time interval, the image is evaluated for importance to obtain an evaluation score for the image. If the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, the image is extracted from the video stream and determined as a candidate image. In this way, on the one hand, a candidate image can be preferentially selected by scene switching, and on the other hand, a candidate image can be selected according to the importance of the image when the scene switching is not detected within a preset time interval, thereby comprehensively determining multiple candidate images based on scene changes and image importance to ensure the quality and information richness of the candidate images. After completing the scene switching detection of each frame of the video stream, the optimal frame image is selected from each candidate image and sent to the server, and then the feedback information determined by the server based on the optimal frame image is determined. By further screening each candidate image to determine the optimal frame image, a more representative and high-quality frame image can be extracted for server-side analysis and processing, which can improve the efficiency of the overall video stream analysis and processing while also improving the accuracy of the video stream analysis and processing, and ensuring the processing quality of the video stream. In addition, the above processing process is completed through the converged communication platform, and the terminal device only needs to upload the video stream to the converged communication platform, which can improve the efficiency and quality of video stream processing while maintaining a certain intelligent cost.

[0095] like Figure 2 As shown, in one embodiment, performing scene switching detection on the image includes:

[0096] S201: Determine whether the image is the first frame image in the video stream.

[0097] S202: If yes, confirm that no scene switch is detected for the image.

[0098] In this step, since scene switching detection needs to use the images of the two preceding and following frames for judgment, if the image is the first frame image in the video stream, it can be determined that no scene switching is detected for the image.

[0099] S203: If not, determining the previous frame image corresponding to the image in the video stream as the reference image.

[0100] If the image is not the first frame image in the video stream, the previous frame image corresponding to the image in the video stream can be used as a reference image, so that the image can be compared with its corresponding reference image to determine whether a scene switch is detected for the image.

[0101] S204: Perform multi-dimensional feature extraction on the image and the reference image respectively to obtain a first feature set corresponding to the image and a second feature set corresponding to the reference image.

[0102] The first feature set includes features of the image in multiple dimensions, and the second feature set includes features of the reference image in multiple dimensions. It should be noted that the dimensions of the first feature set and the second feature set are one-to-one corresponding.

[0103] In this step, the fusion communication platform can perform multi-dimensional feature extraction on the image, and then perform multi-dimensional feature extraction on the reference image, thereby obtaining multi-dimensional features corresponding to the image and multi-dimensional features corresponding to the reference image, and then forming a first feature set based on the multi-dimensional features corresponding to the image, and forming a second feature set based on the multi-dimensional features corresponding to the reference image.

[0104] Furthermore, in the process of performing multi-dimensional feature extraction on the image, multi-dimensional feature extraction can be performed on the image by determining a preset feature extraction model, or by presetting a dimension set, and then performing feature extraction of corresponding dimensions on the image in sequence according to each dimension in the dimension set, so as to complete the feature extraction of multiple dimensions of the image. The specific implementation method can be selected according to actual needs, and this application does not impose specific restrictions on this.

[0105] Specifically, the multi-dimensionality may include but is not limited to color features, texture features, edge features, etc. In one example, if the multi-dimensionality includes color features, texture features, and edge features, the color features may be determined by determining the color histogram of the image, and the texture features and edge features may be determined by using local binary patterns and edge detection algorithms, respectively, thereby obtaining a feature set consisting of the color features, texture features, and edge features of the image.

[0106] S205: Calculate the difference between the first feature set and the second feature set in each dimension respectively to obtain the difference between the first feature set and the second feature set in each dimension.

[0107] The difference degree is used to measure the difference between the first feature set and the second feature set in the corresponding dimension.

[0108] In this step, the difference between the first feature set and the second feature set in each dimension can be calculated respectively according to the difference measurement method corresponding to each dimension to obtain the difference between the first feature set and the second feature set in each dimension.

[0109] In one example, if the multiple dimensions include color features, texture features, and edge features, then when determining a first feature set consisting of the color features, texture features, and edge features of the image and a second feature set consisting of the color features, texture features, and edge features of the reference image, for the color feature dimension, the difference in this dimension can be quantified by calculating the Bhattacharyya distance of the color histogram; for the texture feature dimension, the difference in this dimension can be quantified by calculating the feature similarity difference; for the edge feature dimension, the difference in this dimension can be quantified by calculating the number of changes in edge pixels.

[0110] S206: Obtain the weight of each dimension, and perform weighted summation on the difference between the first feature set and the second feature set in each dimension according to the weight of each dimension to obtain a target difference value.

[0111] The target difference value is used to measure the overall difference between the image and the reference image.

[0112] In this step, the preset weight of each dimension can be obtained first, and then the difference values ​​of the first feature set and the second feature set in each dimension are weighted summed based on the weight of each dimension to obtain the weighted sum value, that is, the target difference value.

[0113] In another embodiment, the converged communication platform may also determine the weight of each dimension according to the importance of each dimension to the detection target in the video stream processing method. This application does not impose any specific restrictions on this.

[0114] S207: Determine whether the target difference value is greater than a preset threshold.

[0115] Specifically, the preset threshold can be set based on experience from a large number of experiments and actual scenarios.

[0116] S208: If the target difference value is greater than a preset threshold, it is confirmed that a scene switch is detected for the image.

[0117] S209: If the target difference value is not greater than the preset threshold, it is determined that no scene switching is detected for the image.

[0118] When the target difference value is greater than the preset threshold, it means that the overall difference between the image and the reference image is large, and it can be confirmed that the scene switch is detected for the image. When the target difference value is not greater than the preset threshold, it means that the overall difference between the image and the reference image is small, and it can be confirmed that the scene switch is not detected for the image.

[0119] In this embodiment, the overall difference between the image and its reference image is used to determine whether a scene switch has occurred for the image, and then determine whether to extract and determine the candidate image. Since the image after the scene switch often contains more information different from the previous image, the image with the scene switch can be located through scene switch detection and extracted as a candidate image, which can ensure the quality of the candidate image and improve the accuracy of the video stream processing and analysis results. In addition, a series of edge computing is performed in the converged communication platform, which can save server resources, shorten the server analysis time, and save bandwidth resources for transmitting data, thereby improving the overall efficiency of video stream processing.

[0120] like Figure 3 As shown, in one embodiment, the importance of the image is evaluated to obtain an evaluation score of the image, including:

[0121] S301: Obtain a preset target detection model.

[0122] Specifically, the target detection model of the present application can be obtained by using a sample image set as a training sample, using the target detection results corresponding to the training sample as a sample label, and using a loss function to train a preset pre-trained model. During the training process, the server can input the training samples and their corresponding sample labels into the pre-trained model for forward propagation, and use the loss function in the back propagation of the pre-trained model to adjust the parameters of the pre-trained model until the pre-trained model meets certain iteration conditions. The pre-trained model that has been trained can be used as a target detection model.

[0123] S302: Perform target detection on the image using a target detection model to obtain a target detection result.

[0124] S303: Based on a preset importance assignment table, an importance evaluation is performed on the target detection result to obtain an importance score of the target detection result in the importance assignment table, and the importance score is determined as an evaluation score of the image.

[0125] The importance assignment table includes multiple targets and the importance score corresponding to each target.

[0126] In this step, when determining the importance assignment table, each target in the target detection result is matched with the importance assignment table to determine the importance score of each target in the target detection result in the importance assignment table, and the sum of the importance scores of each target in the importance assignment table is calculated, that is, the importance score, to complete the importance evaluation of the target detection result and obtain the evaluation score of the image.

[0127] In this embodiment, target recognition and detection are performed on the image based on the target detection model, and then the importance of the target recognition and detection results is evaluated using an importance assignment table to obtain an evaluation score for the image, thereby measuring the degree of match between the information contained in the image and the detection target, so as to decide whether to extract and determine the image as a candidate image, and when no scene change is detected within a preset time period, extraction and determination are performed based on the importance evaluation, which can ensure the rationality of the candidate images and improve the accuracy and processing quality of the final video stream processing results.

[0128] like Figure 4 As shown, in one embodiment, selecting the best frame image from each candidate image and sending the best frame image to the server includes:

[0129] S401: For each candidate image, calculate its edge sharpness based on the candidate image, and determine the clarity of the candidate image according to the edge sharpness.

[0130] In this step, the edge sharpness of the candidate image can be calculated, and then the clarity of the candidate image can be quantified according to the calculated edge sharpness of the candidate image. Generally speaking, a high edge sharpness means a high clarity of the image.

[0131] Furthermore, the edge sharpness of the candidate image can be calculated using algorithms such as gradient method, Laplace operator method, edge detection algorithm, etc. The specific algorithm used can be determined in combination with the actual situation and the characteristics of each algorithm. This application does not make specific restrictions on this.

[0132] S402: Determine the corresponding information entropy according to the grayscale of the candidate image, and measure the correlation between the candidate image and the preset target feature to obtain a correlation quantization value.

[0133] The target feature refers to a feature related to the target detected in the video stream processing method.

[0134] In this step, the information entropy of the candidate image can be determined by determining the grayscale of the candidate image, thereby measuring the amount of information of the candidate image. Then, the preset neural network model can be used to determine whether the candidate image contains the target feature, so as to measure the correlation between the candidate image and the preset target feature and output the correlation quantization value of the candidate image.

[0135] S403: Based on the clarity, information entropy and correlation quantization value of the candidate image, the candidate image is scored to obtain a quality score of the candidate image.

[0136] S404: When the quality score of each candidate image is determined, the candidate image with the highest quality score is selected as the optimal frame image, and the optimal frame image is sent to the server.

[0137] In this embodiment, the quality score of each candidate image is determined by considering the performance of each candidate image in multiple aspects, and then the candidate image with the highest quality score is selected as the optimal frame image. This can save server processing time while ensuring the accuracy of the final analysis and processing results.

[0138] In one embodiment, scoring the candidate image based on the clarity, information entropy, and correlation quantization value of the candidate image to obtain a quality score of the candidate image includes:

[0139] S1: Get the task requirements for this video stream processing.

[0140] The task requirement refers to the requirement for the processing result of this video stream processing.

[0141] S2: Determine the weights of the clarity, information entropy, and correlation quantization value of the candidate image according to task requirements.

[0142] In this step, the requirements for the processing results of this video stream processing can be clarified according to the task requirements, that is, the expected focus on the processing results can be determined, based on which the weights of the clarity, information entropy and correlation quantization value of the candidate image can be determined respectively.

[0143] S3: According to the weights of the clarity, information entropy and correlation quantization value of the candidate image, the clarity, information entropy and correlation quantization value of the candidate image are weighted and summed to obtain the quality score of the candidate image.

[0144] In this embodiment, the comprehensive quality of the candidate images is measured by comprehensively considering multiple factors, and then the quality scores of the candidate images are determined to ensure that the best candidate images can be uploaded to the server for further analysis and processing.

[0145] In one embodiment, determining feedback information determined by the server according to the optimal frame image, and sending the feedback information to the user end includes:

[0146] S1: When the optimal frame image is sent to the server, the server is triggered to match the optimal frame image with the preset black and white list. After obtaining the matching result, the optimal frame image is subjected to structured analysis to obtain the analysis result, and feedback information is generated according to the matching result and the analysis result and sent to the converged communication platform.

[0147] Among them, the blacklist and the whitelist include a blacklist and a whitelist. The blacklist is used to record the characteristics of people or vehicles prohibited from passing, and the whitelist is used to record the characteristics of people or vehicles allowed to pass.

[0148] In this step, when the optimal frame image is sent to the server, the server will match the optimal frame image with the black and white list to generate a matching result, and then perform a structured analysis on the optimal frame image to obtain more information and zinc layer analysis results. For example, for images containing faces, the face analysis algorithm is used to obtain information such as age, gender, and hair color; for images containing license plates, the license plate recognition algorithm is used to obtain information such as license plate number and vehicle color. Finally, feedback information is generated based on the matching results and analysis results and sent to the converged communication platform.

[0149] S2: When feedback information sent by the server is received, the received feedback information is determined as feedback information determined by the server according to the optimal frame image, and the feedback information is sent to each user terminal corresponding to the preset user list.

[0150] In this embodiment, the server performs black and white list matching and structured analysis on the optimal frame image to determine feedback information based on the results of the black and white list matching and the results of the structured analysis. Finally, when the converged communication platform obtains the feedback information, it notifies the corresponding user terminals in the user list of the feedback information.

[0151] In another embodiment, if the fusion communication platform encrypts the optimal frame image when sending it to the server, then when the server receives the optimal frame image, it will decrypt the optimal frame image according to the decryption operation corresponding to the encryption operation, and then perform black and white list matching and structured analysis on the decrypted optimal frame image.

[0152] In one of the embodiments, before sending the optimal frame image to the server, the edge computing-based converged communication video stream processing method further includes:

[0153] The optimal frame image is subjected to noise reduction and size adjustment, and the optimal frame image after noise reduction and size adjustment is encrypted to obtain the encrypted optimal frame image and use it as the optimal frame image finally sent to the server.

[0154] In this embodiment, on the one hand, noise reduction and size adjustment can make the optimal frame image more suitable for processing on the server, and can also improve the accuracy of subsequent structural analysis. On the other hand, encryption of the optimal frame image can improve the security of image transmission.

[0155] In one example, when encrypting the optimal frame image after noise reduction and size adjustment, the AES256 encryption method may be used for encryption.

[0156] In one example, if Figure 5 As shown, Figure 5This is an example diagram of a scenario application of a method for processing a fusion communication video stream based on edge computing provided in an embodiment of the present application. Figure 5 The terminal devices here refer to devices such as drones and cameras that have functions such as video surveillance and recording.

[0157] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] The following describes a fused communication video stream processing device based on edge computing provided in an embodiment of the present application. The fused communication video stream processing device based on edge computing described below and the fused communication video stream processing method based on edge computing described above can refer to each other.

[0159] like Figure 6 As shown, the present application provides a converged communication video stream processing device 500 based on edge computing, which is applied to a converged communication platform, and the device includes:

[0160] The video stream acquisition module 501 is used to acquire the video stream in a unified encoding format;

[0161] A first frame extraction module 502 is used to perform scene switching detection on each frame of the video stream in turn, and if a scene switching is detected, extract the image from the video stream and determine it as a candidate image;

[0162] The second frame extraction module 503 is used to evaluate the importance of the image if no scene switching is detected within a preset time interval to obtain an evaluation score of the image; if the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, the image is extracted from the video stream and determined as a candidate image;

[0163] The image selection module 504 is used to select the best frame image from each candidate image after completing the scene switching detection of each frame image in the video stream, and send the best frame image to the server;

[0164] The information determination module 505 is used to determine the feedback information determined by the server according to the optimal frame image, and send the feedback information to the user end.

[0165] In one embodiment, the first frame extraction module includes:

[0166] A determination submodule, used to determine whether the image is the first frame image in the video stream, and if so, confirm that no scene switching is detected for the image;

[0167] An image determination submodule, for determining, if not, a previous frame of image corresponding to the image in the video stream as a reference image;

[0168] A feature extraction submodule is used to perform multi-dimensional feature extraction on the image and the reference image respectively to obtain a first feature set corresponding to the image and a second feature set corresponding to the reference image;

[0169] A first calculation submodule is used to calculate the difference between the first feature set and the second feature set in each dimension, so as to obtain the difference between the first feature set and the second feature set in each dimension;

[0170] The weighted summation submodule is used to obtain the weight of each dimension, and perform weighted summation on the difference between the first feature set and the second feature set in each dimension according to the weight of each dimension to obtain a target difference value;

[0171] A first confirmation submodule, configured to confirm that a scene switch is detected for the image if the target difference value is greater than a preset threshold;

[0172] The second confirmation submodule is used to confirm that no scene switching is detected for the image if the target difference value is not greater than a preset threshold.

[0173] In one embodiment, the second frame extraction module includes:

[0174] The model acquisition submodule is used to obtain the preset target detection model;

[0175] The target detection submodule is used to perform target detection on the image using the target detection model to obtain the target detection result;

[0176] The score determination submodule is used to perform importance evaluation on the target detection result based on a preset importance assignment table to obtain the importance score of the target detection result in the importance assignment table, and determine the importance score as the evaluation score of the image.

[0177] In one embodiment, the image selection module includes:

[0178] A second calculation submodule, configured to calculate, for each candidate image, an edge sharpness thereof based on the candidate image, and determine a definition of the candidate image according to the edge sharpness;

[0179] A correlation measurement submodule is used to determine the corresponding information entropy according to the grayscale of the candidate image, and to measure the correlation between the candidate image and the preset target feature to obtain a correlation quantization value;

[0180] An image scoring submodule is used to score the candidate image based on the clarity, information entropy and correlation quantization value of the candidate image to obtain a quality score of the candidate image;

[0181] The image selection submodule is used to select the candidate image with the highest quality score as the optimal frame image when determining the quality score of each candidate image, and send the optimal frame image to the server.

[0182] In one embodiment, the image scoring submodule includes:

[0183] A demand acquisition unit is used to obtain the task requirements of this video stream processing;

[0184] A weight determination unit, used to determine the weights of the clarity, information entropy and correlation quantization value of the candidate image according to task requirements;

[0185] The score calculation unit is used to perform weighted summation of the clarity, information entropy and correlation quantization value of the candidate image according to the weights of the clarity, information entropy and correlation quantization value of the candidate image to obtain the quality score of the candidate image.

[0186] In one embodiment, the information determination module includes:

[0187] The list matching submodule is used to trigger the server to match the optimal frame image with a preset black and white list when the optimal frame image is sent to the server, and after obtaining the matching result, perform a structured analysis on the optimal frame image to obtain the analysis result, and generate feedback information according to the matching result and the analysis result and send it to the converged communication platform;

[0188] The information sending submodule is used to determine the received feedback information as feedback information determined by the server according to the optimal frame image when feedback information sent by the server is received, and send the feedback information to each user terminal corresponding to the preset user list.

[0189] In one of the embodiments, the fusion communication video stream processing device based on edge computing further includes:

[0190] The preprocessing module is used to reduce noise and resize the optimal frame image, and encrypt the optimal frame image after noise reduction and resize, obtain the encrypted optimal frame image and use it as the optimal frame image finally sent to the server.

[0191] The division of each module in the above-mentioned fusion communication video stream processing device based on edge computing is only for illustration. In other embodiments, the fusion communication video stream processing device based on edge computing can be divided into different modules as needed to complete all or part of the functions of the above-mentioned fusion communication video stream processing device based on edge computing. Each module in the above-mentioned fusion communication video stream processing device based on edge computing can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0192] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the edge computing-based converged communication video stream processing method as described in any of the above embodiments.

[0193] In one embodiment, the present application also provides a computer device, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the edge computing-based converged communication video stream processing method as described in any of the above embodiments.

[0194] Indicatively, Figure 7 As shown, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 600 may be provided as a server. Figure 7 , the computer device 600 includes a processing component 602, which further includes one or more processors, and a memory resource represented by a memory 601, for storing instructions executable by the processing component 602, such as an application. The application stored in the memory 601 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 602 is configured to execute instructions to perform the converged communication video stream processing method based on edge computing of any of the above embodiments.

[0195] The computer device 600 may further include a power supply component 603 configured to perform power management of the computer device 600, a wired or wireless network interface 604 configured to connect the computer device 600 to a network, and an input / output (I / O) interface 605. The computer device 600 may operate based on an operating system stored in the memory 601, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0196] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0197] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, the singular "one", "one" and "described / the" may also include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "include / comprise" or "have" etc. specify the existence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the existence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0198] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0199] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing fusion communication video stream based on edge computing, characterized in that: Applied to a converged communication platform, the method comprises: Get the video stream in unified encoding format; For each frame of the image in the video stream, scene switching detection is performed on the image in turn, and if a scene switching is detected, the image is extracted from the video stream and determined as a candidate image; If no scene switching is detected within a preset time interval, an importance evaluation is performed on the image to obtain an evaluation score for the image; if the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, the image is extracted from the video stream and determined as a candidate image; After completing the scene switching detection for each frame image in the video stream, selecting the best frame image from each candidate image, and sending the best frame image to the server; Determine feedback information determined by the server according to the optimal frame image, and send the feedback information to the user end.

2. The method for processing fusion communication video stream based on edge computing according to claim 1, characterized in that: The performing scene switching detection on the image includes: Determining whether the image is the first frame image in the video stream, and if so, confirming that no scene switching is detected for the image; If not, determining the previous frame image corresponding to the image in the video stream as the reference image; Performing multi-dimensional feature extraction on the image and the reference image respectively to obtain a first feature set corresponding to the image and a second feature set corresponding to the reference image; Calculating the difference between the first feature set and the second feature set in each dimension respectively to obtain the difference between the first feature set and the second feature set in each dimension; Obtaining the weight of each dimension, and performing weighted summation on the difference between the first feature set and the second feature set in each dimension according to the weight of each dimension to obtain a target difference value; If the target difference value is greater than a preset threshold, it is confirmed that a scene switch is detected for the image; If the target difference value is not greater than the preset threshold, it is determined that no scene switch is detected for the image.

3. The method for processing fusion communication video stream based on edge computing according to claim 1, characterized in that: The step of performing importance assessment on the image to obtain an assessment score of the image includes: Get the preset target detection model; Performing target detection on the image using the target detection model to obtain a target detection result; Based on a preset importance assignment table, the target detection result is evaluated for importance to obtain an importance score of the target detection result on the importance assignment table, and the importance score is determined as the evaluation score of the image.

4. The method for processing fusion communication video stream based on edge computing according to claim 1, characterized in that: The selecting the best frame image from each candidate image and sending the best frame image to the server includes: For each candidate image, calculating its edge sharpness based on the candidate image, and determining the clarity of the candidate image according to the edge sharpness; Determine the corresponding information entropy according to the grayscale of the candidate image, and measure the correlation between the candidate image and the preset target feature to obtain a correlation quantization value; Based on the clarity, information entropy and correlation quantization value of the candidate image, the candidate image is scored to obtain a quality score of the candidate image; When the quality score of each candidate image is determined, the candidate image with the highest quality score is selected as the optimal frame image, and the optimal frame image is sent to the server.

5. The method for processing fusion communication video stream based on edge computing according to claim 4 is characterized in that: Scoring the candidate image based on the clarity, information entropy, and correlation quantization value of the candidate image to obtain a quality score of the candidate image includes: Get the task requirements for this video stream processing; Determine the clarity, information entropy and weights of the correlation quantization value of the candidate image respectively according to the task requirements; According to the weights of the clarity, information entropy and correlation quantization value of the candidate image, the clarity, information entropy and correlation quantization value of the candidate image are weighted and summed to obtain the quality score of the candidate image.

6. The method for processing fusion communication video stream based on edge computing according to claim 1, characterized in that: The step of determining feedback information determined by the server according to the optimal frame image, and sending the feedback information to the user end includes: When the optimal frame image is sent to the server, the server is triggered to match the optimal frame image with a preset black and white list, and after obtaining a matching result, a structured analysis is performed on the optimal frame image to obtain an analysis result, and feedback information is generated according to the matching result and the analysis result and sent to the converged communication platform; When the feedback information sent by the server is received, the received feedback information is determined as feedback information determined by the server according to the optimal frame image, and the feedback information is sent to each user terminal corresponding to the preset user list.

7. The method for processing fusion communication video stream based on edge computing according to any one of claims 1 to 6, characterized in that: Before sending the optimal frame image to the server, the method further includes: The optimal frame image is subjected to noise reduction and size adjustment, and the optimal frame image after noise reduction and size adjustment is encrypted to obtain the encrypted optimal frame image and use it as the optimal frame image finally sent to the server.

8. A fusion communication video stream processing device based on edge computing, characterized in that: Applied to a converged communication platform, the device comprises: A video stream acquisition module is used to acquire video streams in a unified encoding format; A first frame extraction module is used to perform scene switching detection on each frame of the image in the video stream in turn, and if a scene switching is detected, extract the image from the video stream and determine it as a candidate image; A second frame extraction module is used to evaluate the importance of the image if no scene switching is detected within a preset time interval to obtain an evaluation score for the image; if the evaluation score is greater than a first preset threshold and the number of image frames extracted within a time period corresponding to the preset time interval does not reach a second preset threshold, extract the image from the video stream and determine it as a candidate image; An image selection module, configured to select an optimal frame image from each candidate image after completing scene switching detection for each frame image in the video stream, and send the optimal frame image to a server; The information determination module is used to determine the feedback information determined by the server according to the optimal frame image, and send the feedback information to the user end.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the edge computing-based converged communication video stream processing method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the edge computing-based converged communication video stream processing method as described in any one of claims 1 to 7 are performed.

Images