Mosaic severity detection method and image processing method and device

Abstract

The embodiment of the invention relates to a mosaic severity detection method and device and an image processing method and device.The mosaic severity detection method comprises the steps that outflow data of a streaming media picture collected by a security camera in real time are obtained, and the outflow data are picture data representing the streaming media picture in the transmission process; based on the streaming data, determining the picture complexity of the streaming media picture, the picture complexity representing the complexity of the streaming media picture; and based on the picture complexity, determining the mosaic severity of the streaming media picture. Therefore, when the streaming media picture data is output, the mosaic severity can be detected, and the timeliness of detecting the mosaic severity of the streaming media picture is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of security camera, and particularly relates to a mosaic severity detection method, an image processing method and device. BACKGROUND

[0002] In the related art, in the processing of a stream media picture collected by a security camera in real time, mosaic phenomenon may occur due to many factors, that is, pixel missing areas in the form of randomly distributed blocks of different sizes and positions appear in the stream media picture. The mosaic phenomenon seriously affects the quality and observability of the stream media picture, and therefore, a method is needed to detect the mosaic severity of the stream media picture.

[0003] In the prior art, mosaic in a displayed picture is identified by human observation after the picture is displayed, and then the mosaic severity of the picture is determined according to the identified mosaic.

[0004] However, in the above solution, the mosaic severity detection of the stream media picture is not timely. SUMMARY

[0005] In view of this, to solve some or all of the above technical problems, the present application provides a mosaic severity detection method, an image processing method and device.

[0006] In a first aspect, the present application provides a mosaic severity detection method, which comprises:

[0007] obtaining outflow data of a stream media picture collected by a security camera in real time, wherein the outflow data is picture data representing the stream media picture in a transmission process;

[0008] determining a picture complexity of the stream media picture based on the outflow data, wherein the picture complexity represents a complexity of the stream media picture;

[0009] determining a mosaic severity of the stream media picture based on the picture complexity.

[0010] In one possible implementation, the picture complexity comprises a spatial complexity of the stream media picture, the stream media picture comprises at least two picture blocks, and the spatial complexity represents a difference degree between the picture blocks of the stream media picture; and

[0011] The determining of the picture complexity of the stream media picture based on the outflow data comprises:

[0012] determining a spatial complexity of the stream media picture based on the outflow data; and

[0013] The determining of the mosaic severity of the stream media picture based on the picture complexity comprises:

[0014] The determining of the mosaic severity of the stream media picture based on the spatial complexity.

[0015] In a possible implementation, the determining of the spatial complexity of the stream media picture based on the outflow data comprises:

[0016] The dividing of the stream media picture into the first set of picture blocks based on the outflow data;

[0017] The determining of a residual between a first picture block in the first set of picture blocks and a first reference block, wherein the first reference block is a picture block in the stream media picture;

[0018] The determining of the spatial complexity of the stream media picture based on the determined residual.

[0019] In a possible implementation, the picture complexity comprises a temporal complexity of the stream media picture, the temporal complexity representing a difference degree between a history picture of the stream media picture and the stream media picture; and

[0020] The determining of the picture complexity of the stream media picture based on the outflow data comprises:

[0021] The determining of the temporal complexity of the stream media picture based on the outflow data; and

[0022] The determining of the mosaic severity of the stream media picture based on the picture complexity comprises:

[0023] The determining of the mosaic severity of the stream media picture based on the temporal complexity.

[0024] In a possible implementation, the determining of the temporal complexity of the stream media picture based on the outflow data comprises:

[0025] The dividing of the stream media picture into a second set of picture blocks based on the outflow data;

[0026] The determining of a motion vector of a second picture block in the second set of picture blocks based on a second reference block, wherein the second reference block is a picture block in the history picture;

[0027] The determining of the temporal complexity of the stream media picture based on the determined motion vector.

[0028] In a possible implementation, the second reference block is determined in the following manner:

[0029] determining a third picture block set obtained by dividing the history picture;

[0030] from the third picture block set, determining third picture blocks that are time or spatial neighbors of a second picture block in the streaming picture, to obtain a picture block subset;

[0031] determining a motion vector of each third picture block in the picture block subset, to obtain a motion vector set, wherein the third picture blocks in the picture block subset correspond to the motion vectors in the motion vector set in a one-to-one manner;

[0032] determining a rate-distortion cost of each motion vector in the motion vector set, to obtain a rate-distortion cost set, wherein the rate-distortion costs in the rate-distortion cost set correspond to the third picture blocks in the picture block subset in a one-to-one manner;

[0033] determining, as the second reference block, a picture block corresponding to a rate-distortion cost with a minimum value in the rate-distortion cost set.

[0034] In a possible implementation, the determining the picture complexity of the streaming picture based on the outflow data comprises:

[0035] determining a first complexity of the streaming picture based on the outflow data by using a first algorithm, wherein the first algorithm is used to determine a complexity of the streaming picture based on the outflow data, and the first complexity represents the complexity of the streaming picture determined by using the first algorithm;

[0036] determining a second complexity of the streaming picture based on the outflow data by using a second algorithm, wherein the second algorithm is used to determine a complexity of the streaming picture based on the outflow data, the second algorithm is different from the first algorithm, and the second complexity represents the complexity of the streaming picture determined by using the second algorithm;

[0037] determining the picture complexity of the streaming picture based on the first complexity and the second complexity.

[0038] In a second aspect, an embodiment of the present application provides an image processing method, and the method comprises:

[0039] determining a mosaic severity of a streaming picture by using a preset method, wherein the preset method is the method for detecting the mosaic severity described in the first aspect;

[0040] determining a code rate of the streaming picture based on the mosaic severity.

[0041] transmitting the stream-out data of the stream media picture according to the code rate.

[0042] In a possible implementation, after the method of determining the mosaic severity of the stream media picture by using the preset method, the method further includes:

[0043] determining whether the mosaic severity is greater than or equal to a preset degree threshold;

[0044] generating prompt information indicating that the stream media picture has a severe mosaic, in a case where the mosaic severity is greater than or equal to the preset degree threshold.

[0045] In a third aspect, an embodiment of the present application provides a device for detecting a mosaic severity, and the device includes:

[0046] an acquisition unit, configured to acquire stream-out data of a stream media picture collected by a security camera in real time, wherein the stream-out data is picture data representing the stream media picture in a transmission process;

[0047] a first determination unit, configured to determine a picture complexity of the stream media picture based on the stream-out data, wherein the picture complexity represents a complexity of the stream media picture;

[0048] a second determination unit, configured to determine a mosaic severity of the stream media picture based on the picture complexity.

[0049] In a possible implementation, the picture complexity includes a spatial complexity of the stream media picture, the stream media picture includes at least two picture blocks, and the spatial complexity represents a difference degree between the picture blocks of the stream media picture.

[0050] The determination of the picture complexity of the stream media picture based on the stream-out data includes:

[0051] determining a spatial complexity of the stream media picture based on the stream-out data; and

[0052] The determination of the mosaic severity of the stream media picture based on the picture complexity includes:

[0053] determination of the mosaic severity of the stream media picture based on the spatial complexity.

[0054] In a possible implementation, the determination of the spatial complexity of the stream media picture based on the stream-out data includes:

[0055] dividing the stream media picture into a first picture block set based on the stream-out data.

[0056] determining a residual between a first picture block in the first picture block set and a first reference block, wherein the first reference block is located in the first picture block in the streaming picture;

[0057] determining a spatial complexity of the streaming picture based on the determined residual.

[0058] In one possible implementation, the picture complexity includes a temporal complexity of the streaming picture, the temporal complexity representing a degree of difference between a history picture of the streaming picture and the streaming picture; and

[0059] The determining the picture complexity of the streaming picture based on the outflow data includes:

[0060] determining the temporal complexity of the streaming picture based on the outflow data; and

[0061] The determining the mosaic severity of the streaming picture based on the picture complexity includes:

[0062] determining the mosaic severity of the streaming picture based on the temporal complexity.

[0063] In one possible implementation, the determining the temporal complexity of the streaming picture based on the outflow data includes:

[0064] dividing the streaming picture into a second picture block set based on the outflow data;

[0065] determining a motion vector of a second picture block in the second picture block set based on a second reference block, wherein the second reference block is located in the second picture block in the history picture;

[0066] determining the temporal complexity of the streaming picture based on the determined motion vector.

[0067] In one possible implementation, the second reference block is determined in the following manner:

[0068] determining a third picture block set obtained by dividing the history picture;

[0069] determining a third picture block that is time or spatial adjacent to the second picture block in the streaming picture from the third picture block set to obtain a picture block subset;

[0070] determine a motion vector of each third picture block in the picture block subset to obtain a motion vector set, wherein the third picture block in the picture block subset and the motion vector in the motion vector set are in one-to-one correspondence;

[0071] determine a rate-distortion cost of the motion vector in the motion vector set to obtain a rate-distortion cost set, wherein the rate-distortion cost in the rate-distortion cost set and the third picture block in the picture block subset are in one-to-one correspondence;

[0072] determine the picture block corresponding to the rate-distortion cost with the minimum value in the rate-distortion cost set as the second reference block.

[0073] In one possible implementation, the determining the picture complexity of the streaming media picture based on the outflow data comprises:

[0074] determining a first complexity of the streaming media picture based on the outflow data by using a first algorithm, wherein the first algorithm is used to determine the complexity of the streaming media picture based on the outflow data, and the first complexity represents the complexity of the streaming media picture determined by using the first algorithm;

[0075] determining a second complexity of the streaming media picture based on the outflow data by using a second algorithm, wherein the second algorithm is used to determine the complexity of the streaming media picture based on the outflow data, the second algorithm is different from the first algorithm, and the second complexity represents the complexity of the streaming media picture determined by using the second algorithm;

[0076] determining the picture complexity of the streaming media picture based on the first complexity and the second complexity.

[0077] In a fourth aspect, an embodiment of the present application provides an image processing apparatus, and the apparatus comprises:

[0078] a third determining unit configured to determine the mosaic severity of the streaming media picture by using a preset apparatus, wherein the preset apparatus is the method for detecting the mosaic severity described in the first aspect;

[0079] a fourth determining unit configured to determine the code rate of the streaming media picture based on the mosaic severity;

[0080] a transmitting unit configured to transmit the outflow data of the streaming media picture according to the code rate.

[0081] In one possible implementation, after the determining the mosaic severity of the streaming media picture by using the preset apparatus, the apparatus further comprises:

[0082] The fifth determining unit is configured to determine whether the mosaic severity is greater than or equal to a preset degree threshold.

[0083] The generating unit is configured to generate prompt information indicating that the mosaic of the stream media picture is severe, when the mosaic severity is greater than or equal to the preset degree threshold.

[0084] In a fifth aspect, an electronic device is provided, and the electronic device comprises:

[0085] A memory is configured to store a computer program.

[0086] A processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method for detecting the mosaic severity of the first aspect or the method of any one of the embodiments of the image processing method of the second aspect is implemented.

[0087] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the method for detecting the mosaic severity of the first aspect or the method of any one of the embodiments of the image processing method of the second aspect is implemented.

[0088] In a fifth aspect, a computer program product is provided, and the computer program product comprises computer readable code. When the computer readable code is executed on a device, the processor in the device implements the method for detecting the mosaic severity of the first aspect or the method of any one of the embodiments of the image processing method of the second aspect.

[0089] The method for detecting the mosaic severity provided in the embodiments of the present application can obtain outflow data of a stream media picture collected by a security camera in real time, wherein the outflow data is picture data representing the stream media picture in a transmission process. Then, based on the outflow data, the picture complexity of the stream media picture is determined, wherein the picture complexity represents the complexity of the stream media picture. Then, based on the picture complexity, the mosaic severity of the stream media picture is determined. In this way, the mosaic severity can be detected when the stream media picture data is outflowed, and the timeliness of the detection of the mosaic severity of the stream media picture is improved.

[0090] The image processing method provided in the embodiments of the present application can determine the mosaic severity of a streaming media picture by using a preset method, wherein the preset method is the method for detecting the mosaic severity described in the first aspect, then, the code rate of the streaming media picture is determined based on the mosaic severity, and then the outflow data of the streaming media picture is transmitted according to the code rate. In this way, different code rates of the streaming media picture can be determined based on different mosaic severities, thereby improving the matching degree between the transmission code rate of the streaming media picture and the mosaic severity thereof. BRIEF DESCRIPTION OF DRAWINGS

[0091] The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0092] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0093] One or more embodiments are exemplarily illustrated by pictures in the drawings corresponding thereto, and these exemplary illustrations do not constitute a limitation on the embodiments, and elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified, and the drawings do not constitute a proportional limitation.

[0094] Figure 1 A flowchart of a method for detecting mosaic severity provided in the embodiments of the present application;

[0095] Figure 2 A flowchart of another method for detecting mosaic severity provided in the embodiments of the present application;

[0096] Figure 3 A flowchart of a method for detecting mosaic severity provided in the embodiments of the present application;

[0097] Figure 4 A flowchart of another method for detecting mosaic severity provided in the embodiments of the present application;

[0098] Figure 5 A flowchart of a method for detecting mosaic severity provided in the embodiments of the present application;

[0099] Figure 6 A structural diagram of a device for detecting mosaic severity provided in the embodiments of the present application;

[0100] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0101] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0102] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this application.

[0103] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of this application are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor do they indicate the logical order between them.

[0104] It should also be understood that in this embodiment, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0105] It should also be understood that any component, data or structure mentioned in the embodiments of this application can generally be understood as one or more unless explicitly defined or given contrary guidance in the context.

[0106] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects have an "or" relationship.

[0107] It should also be understood that the description of the various embodiments in this application emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0108] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0109] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0110] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0111] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. To facilitate understanding of the embodiments of this application, the application will be described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0112] To address the technical problem of low timeliness in detecting the severity of mosaic in streaming media images in existing technologies, this application provides a method for detecting the severity of mosaic, an image processing method, and an apparatus, which can improve the timeliness of detecting the severity of mosaic in streaming media images.

[0113] Figure 1 This is a flowchart illustrating a method for detecting the severity of mosaic artifacts, provided in an embodiment of this application. This method can be applied to one or more electronic devices such as smartphones, laptops, desktop computers, portable computers, and servers. Furthermore, the execution entity of this method can be hardware or software. When the execution entity is hardware, it can be one or more of the aforementioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the execution entity is software, this method can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are imposed here.

[0114] like Figure 1 As shown, the method specifically includes:

[0115] Step 101: Obtain the outgoing data of the streaming media image captured in real time by the security camera, wherein the outgoing data is the image data representing the streaming media image during transmission.

[0116] In this embodiment, the streaming media image can refer to an image transmitted and displayed using streaming media technology. Here, the streaming media image can be a streaming media image captured in real time by a security camera.

[0117] Outgoing stream data can represent the image data of a streaming media image during the transmission of the streaming media image from one device to another. Outgoing stream data can include at least one of the following: image data such as the position, color, brightness, chroma, number of elements, and type of elements in the streaming media image (e.g., pixels, people, animals, trees, etc.).

[0118] As an example, the outgoing data could be the real-time streaming video data generated during the transmission of data from the security camera to other devices (such as display devices).

[0119] Step 102: Based on the outflow data, determine the image complexity of the streaming media image, wherein the image complexity represents the degree of complexity of the streaming media image.

[0120] In this embodiment, various methods can be used to determine the image complexity of the streaming media image based on the outgoing data.

[0121] As an example, the outgoing data can be input into a pre-trained picture complexity determination model to obtain the picture complexity of the streaming media picture.

[0122] The image complexity determination model can represent the correspondence between the image complexity of streaming data and streaming video images. This model can be obtained using machine learning algorithms, trained on training samples containing outgoing streaming data and image complexity.

[0123] In addition, other methods can be used to determine the video complexity of the streaming media image based on the outgoing data. Please refer to the description below for details, which will not be elaborated upon here.

[0124] Step 103: Determine the severity of mosaic in the streaming media image based on the image complexity.

[0125] In this embodiment, the severity of the mosaic can represent the number of mosaics in the streaming media image, the ratio of the number of mosaics in the streaming media image to the number of pixels in the streaming media image, or the severity level of the mosaic in the streaming media image.

[0126] In some optional implementations of this embodiment, the image complexity includes the spatial complexity of the streaming media image, which includes at least two image blocks, and the spatial complexity represents the degree of difference between the image blocks of the streaming media image.

[0127] Based on this, the following method can be used to determine the picture complexity of the streaming media image based on the outflow data: the spatial complexity of the streaming media image based on the outflow data.

[0128] As an example, the spatial complexity of the streaming media image can be determined based on the outgoing data in the following manner:

[0129] When the outgoing data includes the number and types of elements in the streaming media image, the spatial complexity of the streaming media image can be determined by the number and types of elements in the image. These elements include graphics, lines, and colored areas. For example, the spatial complexity of the streaming media image can be positively correlated with both the number and types of elements. Specifically, an image with only a few simple geometric shapes has lower spatial complexity, while an image containing many objects of different shapes, rich colors, and numerous details has higher spatial complexity.

[0130] Here, we can also consider the combinations and relationships between elements. If the arrangement and combination of elements are simple and regular, then the space complexity is relatively low; conversely, if the relationships between elements are complex, intertwined, and without obvious patterns, then the space complexity will increase.

[0131] In addition, other methods can be used to determine the spatial complexity of the streaming media image based on the outflow data. Please refer to the following description for details, which will not be elaborated here.

[0132] Furthermore, the severity of mosaic in the streaming media image can be determined based on the image complexity as follows: the severity of mosaic in the streaming media image is determined based on the spatial complexity.

[0133] Here, the severity of mosaic in the streaming media image can be determined based solely on the spatial complexity, or it can be determined based on both the spatial complexity and the time complexity.

[0134] The specific meaning of time complexity will be described later and will not be elaborated here.

[0135] As an example, the severity of mosaic artifacts in the streaming media image can be determined in several ways based on the aforementioned spatial complexity.

[0136] For example, the severity of mosaic in the streaming media image can be determined based on the spatial complexity using a pre-defined correspondence table, formula, or pre-fitted curve.

[0137] In some cases, the severity of pixelation in streaming media can be positively correlated with spatial complexity. For example, the higher the spatial complexity, the more severe the pixelation in the streaming media can be.

[0138] In addition, various methods can be used to determine the severity of mosaic in the streaming media image based on the aforementioned space complexity and time complexity.

[0139] For example, the severity of mosaic in the streaming media image can be determined based on the space complexity and time complexity using a pre-defined correspondence table, formula, or pre-fitted curve.

[0140] In practice, curves representing space complexity and mosaic severity, as well as curves representing time complexity and mosaic severity, can be fitted separately. If the space complexity of the streaming video is greater than or equal to its time complexity, the mosaic severity corresponding to that space complexity can be used as the final determined mosaic severity of the streaming video. If the space complexity of the streaming video is less than its time complexity, the mosaic severity corresponding to that time complexity can be used as the final determined mosaic severity of the streaming video.

[0141] In some cases, the severity of pixelation in streaming media can be positively correlated with both spatial and temporal complexity. For example, higher spatial complexity generally leads to more severe pixelation, as does higher temporal complexity.

[0142] It is understandable that among the above optional implementation methods, the severity of mosaic can be detected by the spatial complexity of the streaming media image when the streaming media image data is outgoing, thereby improving the accuracy of mosaic severity detection in the streaming media image.

[0143] In some application scenarios of the above-mentioned optional implementation methods, the spatial complexity of the streaming media image can also be determined based on the outgoing data in the following manner:

[0144] The first step is to divide the streaming media image into a first set of image blocks based on the outgoing data.

[0145] The first set of picture blocks can be a set of picture blocks (i.e., first picture blocks) obtained by dividing the streaming media picture.

[0146] Here, since the outgoing data includes the position information of each pixel in the streaming media image, the streaming media image can be divided into multiple first frame blocks based on the outgoing data. For example, the streaming media image can be divided into 16×16 first frame blocks, or it can be divided into 8×8 first frame blocks.

[0147] The second step is to determine the residual between the first frame block and the first reference block for the first frame block in the first frame block set.

[0148] The first reference block is a first frame block located in the streaming media frame. The first reference block can be used to determine the spatial complexity of the streaming media frame.

[0149] Here, each first frame block in the first frame block set can be traversed to determine the residual between each first frame block and the first reference block. In other words, the calculated residual corresponds one-to-one with the first frame blocks in the first frame block set.

[0150] The residual between the first frame block and the first reference block can represent the difference between the first frame block and the first reference block.

[0151] Each first frame tile in the first frame tile set can correspond to a first reference tile. The first reference tiles corresponding to different first frame tiles can be the same or different.

[0152] Here, multiple methods can be used to determine the first reference block corresponding to the first screen block A.

[0153] As an example, the first frame block adjacent to first frame block A in the first frame block set can be identified as the first reference block corresponding to first frame block A. In this case, the two first reference blocks corresponding to two different first frame blocks can be the same or different.

[0154] As another example, the first frame block M located at the center of the streaming media frame in the first frame block set can also be defined as the first reference block corresponding to the first frame block A. In this case, the two first reference blocks corresponding to the two different first frame blocks are the same.

[0155] The third step is to determine the spatial complexity of the streaming media image based on the determined residual.

[0156] As an example, the spatial complexity of the streaming media image can be obtained by first applying the Hadamard transform to the determined residuals, and then calculating the sum of the absolute values ​​of the transformed residuals.

[0157] As another example, the spatial complexity of the streaming video can also be obtained by summing the absolute values ​​of the determined residuals.

[0158] It is understandable that, in the above application scenarios, the spatial complexity of a streaming media frame can be determined by the residuals between each frame block and its reference block. This improves the accuracy of determining the spatial complexity of the streaming media frame.

[0159] In some optional implementations of this embodiment, the image complexity includes the time complexity of the streaming media image, which represents the degree of difference between the historical images of the streaming media image and the streaming media image itself.

[0160] Based on this, the following method can be used to determine the picture complexity of the streaming media image based on the outflow data: the time complexity of the streaming media image based on the outflow data.

[0161] As an example, the time complexity of the streaming media frame can be determined based on the outgoing data in the following manner:

[0162] The outflow data is input into a pre-trained time complexity determination model to obtain the time complexity of the streaming media image.

[0163] The aforementioned time complexity determination model can represent the correspondence between streaming data and time complexity. This model can be obtained using machine learning algorithms, trained on training samples containing outgoing data and time complexity values.

[0164] In addition, other methods can be used to determine the time complexity of the streaming media image based on the outflow data. Please refer to the following description for details, which will not be elaborated here.

[0165] Furthermore, the severity of mosaic artifacts in the streaming media image can be determined based on the image complexity as follows: the severity of mosaic artifacts in the streaming media image is determined based on the time complexity.

[0166] Here, the severity of mosaic in the streaming media image can be determined based solely on the time complexity, or it can be determined based on both the space complexity and the time complexity.

[0167] As an example, the severity of pixelation in the streaming media image can be determined in several ways based on the time complexity.

[0168] For example, the severity of pixelation in the streaming media image can be determined based on the time complexity using a pre-defined correspondence table, formula, or pre-fitted curve.

[0169] In some cases, the severity of pixelation in streaming media can be positively correlated with time complexity. For example, the higher the time complexity, the more severe the pixelation in the streaming media can be.

[0170] In addition, various methods can be used to determine the severity of mosaic in the streaming media image based on the aforementioned space complexity and time complexity.

[0171] For example, the severity of mosaic in the streaming media image can be determined based on the space complexity and time complexity using a pre-defined correspondence table, formula, or pre-fitted curve.

[0172] In practice, curves representing space complexity and mosaic severity, as well as curves representing time complexity and mosaic severity, can be fitted separately. If the space complexity of the streaming video is greater than or equal to its time complexity, the mosaic severity corresponding to that space complexity can be used as the final determined mosaic severity of the streaming video. If the space complexity of the streaming video is less than its time complexity, the mosaic severity corresponding to that time complexity can be used as the final determined mosaic severity of the streaming video.

[0173] In some cases, the severity of pixelation in streaming media can be positively correlated with both spatial and temporal complexity. For example, higher spatial complexity generally leads to more severe pixelation, as does higher temporal complexity.

[0174] It is understandable that among the above optional implementation methods, the severity of mosaic can be detected by the time complexity of the streaming media image when the streaming media image data is outgoing, thereby improving the accuracy of mosaic severity detection in the streaming media image.

[0175] In some application scenarios of the above-mentioned optional implementation methods, the time complexity of the streaming media image can be determined based on the outgoing data in the following manner:

[0176] The first step is to divide the streaming media image into a second set of image blocks based on the outgoing data.

[0177] The second set of screen blocks can be a set of screen blocks (i.e., second screen blocks) obtained by dividing the streaming media screen.

[0178] Here, since the outgoing data includes the position information of each pixel in the streaming media image, the streaming media image can be divided into multiple image blocks based on the outgoing data. For example, the streaming media image can be divided into 16×16 second image blocks, or it can be divided into 8×8 second image blocks.

[0179] The number of second frames contained in the second frame set may be equal to or different from the number of first frames contained in the first frame set.

[0180] The methods for dividing the streaming media frame to obtain the second frame block set and the first frame block set can be the same or different.

[0181] The second step is to determine the motion vector of the second frame block in the second frame block set based on the second reference block.

[0182] The second reference block is a second frame block located within the historical frame. The second reference block can be used to determine the time complexity of the streaming media frame.

[0183] Here, each second frame in the second frame set can be traversed to determine the motion vector of each second frame based on the second reference frame. In other words, the calculated motion vectors correspond one-to-one with the second frames in the second frame set.

[0184] The historical frame can be any frame of streaming media preceding the streaming media frame. For example, the historical frame can be the frame preceding the streaming media frame.

[0185] Each second frame in the second frame set can correspond to a second reference frame. The second reference frames corresponding to different second frame frames can be the same or different.

[0186] Here, multiple methods can be used to determine the second reference block corresponding to the second screen block B.

[0187] As an example, the second frame block obtained by dividing the historical frame can be identified as the second reference block corresponding to the second frame block B, which has the same image coordinates as the second frame block B. In this case, the two second reference blocks corresponding to the two different second frame blocks are different.

[0188] Alternatively, other methods can be used to determine the second reference block corresponding to the second screen block B. Please refer to the description below for details, which will not be elaborated upon here.

[0189] The third step is to determine the time complexity of the streaming media image based on the determined motion vectors.

[0190] Here, the time complexity of the streaming media image can be determined in various ways based on the determined motion vector.

[0191] As an example, the value of the determined motion vector can be used to determine the time complexity of the streaming video.

[0192] As another example, the time complexity of the streaming media image can also be determined based on the determined motion vector by using a preset correspondence table, formula, or pre-fitted curve.

[0193] It is understandable that, in the above application scenarios, the time complexity of the streaming media frame can be determined by the motion vectors of each frame in the streaming media frame. This can improve the accuracy of determining the time complexity of the streaming media frame.

[0194] In some of the above application scenarios, the second reference block can be determined in the following way:

[0195] The first step is to determine the set of third-screen blocks obtained from the division of the historical screen.

[0196] The third set of screen blocks can be a set of screen blocks (i.e., third screen blocks) obtained by dividing the historical screen.

[0197] The second step is to determine, from the set of third frame blocks, the third frame blocks that are temporally or spatially adjacent to the second frame blocks in the streaming media frame, so as to obtain a subset of frame blocks.

[0198] Among them, the third frame block 2, which is temporally adjacent to the second frame block 1 in the streaming media, can satisfy the following: the image coordinates of the third frame block 2 in the above-mentioned historical frame are the same as the image coordinates of the second frame block 1 in the streaming media.

[0199] The third frame block 3, which is spatially adjacent to the second frame block 1 in the streaming media image, can satisfy the following condition: the position of the third frame block 3 in the streaming media image is adjacent to the position of the second frame block 1 in the streaming media image.

[0200] The subset of picture blocks may include at least one of the following: a third picture block that is temporally adjacent to the second picture block in the streaming media picture, and a third picture block that is spatially adjacent to the second picture block in the streaming media picture.

[0201] The third step is to determine the motion vectors of each third frame block in the frame block subset to obtain a set of motion vectors.

[0202] The third frame block in the frame block subset corresponds one-to-one with the motion vector in the motion vector set.

[0203] A motion vector can represent the displacement of a third frame block within a frame block subset relative to its corresponding reference block. Motion vectors can be used to describe the positional changes of a third frame block within a frame block subset relative to a reference block.

[0204] Motion vectors are typically defined as displacements in the horizontal and vertical directions.

[0205] For example, a motion vector of (5, -3) indicates that the device has moved 5 pixels to the right horizontally and 3 pixels upward vertically.

[0206] The fourth step is to determine the rate-distortion cost of the motion vectors in the set of motion vectors to obtain the rate-distortion cost set.

[0207] The rate-distortion cost in the rate-distortion cost set corresponds one-to-one with the third frame block in the frame block subset.

[0208] Here, the rate-distortion cost of the motion vectors in the set of motion vectors can be determined in the following way:

[0209] First, determine the distortion metric. Common distortion metrics include mean squared error (MSE) and peak signal-to-noise ratio (PSNR).

[0210] For motion vectors, distortion is typically calculated based on prediction error. Prediction error refers to the difference between the predicted pixel and the actual pixel obtained by predicting using the currently calculated motion vectors.

[0211] Then, consider the number of bits required to encode the motion vector, i.e., the code rate.

[0212] The formula for calculating rate-distortion cost can usually be expressed as:

[0213] Cost = Distortion + λ × Rate

[0214] Here, Distortion is the distortion calculated based on the prediction error mentioned above, Rate is the number of bits required for motion vector coding, and λ is a Lagrange multiplier used to balance the importance of distortion and bit rate.

[0215] The fifth step is to determine the image block corresponding to the smallest rate-distortion cost in the rate-distortion cost set as the second reference block.

[0216] It is understandable that, in the above situation, by determining the third frame block corresponding to the smallest rate distortion cost as the second reference block corresponding to the second frame block in the streaming media frame, the visual difference between the displayed streaming media frame and the original streaming media frame can be reduced, thereby improving the visual quality.

[0217] The mosaic severity detection method provided in this application can acquire outgoing streaming data of a security camera's real-time captured streaming media image. This outgoing data represents the image data of the streaming media image during transmission. Then, based on the outgoing data, the image complexity of the streaming media image is determined, where image complexity represents the overall complexity of the streaming media image. Finally, based on the image complexity, the mosaic severity of the streaming media image is determined. Therefore, mosaic severity can be detected during the outgoing streaming media image data, improving the timeliness of mosaic severity detection.

[0218] Figure 2 This is a flowchart illustrating another method for detecting the severity of mosaic artifacts provided in an embodiment of this application. Figure 2 As shown, the method specifically includes:

[0219] Step 201: Obtain the outflow data of the streaming media image captured in real time by the security camera, wherein the outflow data is the image data representing the streaming media image during the transmission process.

[0220] In this embodiment, step 201 and Figure 1 Step 101 in the corresponding embodiment is basically the same, and will not be repeated here.

[0221] Step 202: Using a first algorithm, determine the first complexity of the streaming media image based on the outgoing data, wherein the first algorithm is used to determine the complexity of the streaming media image based on the outgoing data, and the first complexity represents the complexity of the streaming media image determined by the first algorithm.

[0222] In this embodiment, the first algorithm can be an algorithm for determining the time complexity of the streaming media image, or it can be an algorithm for determining the space complexity of the streaming media image.

[0223] Step 203: Using a second algorithm, a second complexity of the streaming media image is determined based on the outgoing data. The second algorithm is used to determine the complexity of the streaming media image based on the outgoing data. The second algorithm is different from the first algorithm. The second complexity represents the complexity of the streaming media image determined by the second algorithm.

[0224] In this embodiment, the second algorithm can be an algorithm for determining the time complexity of the streaming media image, or it can be an algorithm for determining the space complexity of the streaming media image.

[0225] When the first algorithm is an algorithm for determining the time complexity of the streaming media frame, the second algorithm can be an algorithm for determining the space complexity of the streaming media frame. When the first algorithm is an algorithm for determining the space complexity of the streaming media frame, the second algorithm can be an algorithm for determining the time complexity of the streaming media frame.

[0226] In this embodiment, the specific methods for determining the time complexity and space complexity of the streaming media image can be found in the description above, and will not be repeated here.

[0227] Step 204: Based on the first complexity and the second complexity, determine the picture complexity of the streaming media picture, wherein the picture complexity represents the degree of complexity of the streaming media picture.

[0228] In this embodiment, the image complexity of the streaming media image can be determined in a variety of ways based on the first complexity and the second complexity.

[0229] As an example, when the first complexity and the second complexity are represented numerically, the weighted sum of the first complexity and the second complexity can be used to determine the picture complexity of the streaming media picture.

[0230] As another example, if the first complexity is greater than or equal to the second complexity, the first complexity can be determined as the picture complexity of the streaming media image; if the first complexity is less than the second complexity, the second complexity can be determined as the picture complexity of the streaming media image.

[0231] Step 205: Determine the severity of mosaic in the streaming media image based on the image complexity.

[0232] In this embodiment, step 205 and Figure 1 Step 103 in the corresponding embodiment is basically the same, and will not be repeated here.

[0233] It should be noted that, in addition to the contents described above, this embodiment may also include... Figure 1 The corresponding technical features described in the corresponding embodiments, thereby achieving Figure 1 For details on the technical effectiveness of the mosaic severity detection method shown, please refer to [link / reference]. Figure 1 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0234] The mosaic severity detection method provided in this application uses two different algorithms to determine the image complexity of the streaming media image from at least two dimensions, thereby improving the accuracy of the final determined image complexity of the streaming media image.

[0235] Figure 3 This is a flowchart illustrating an image processing method provided in an embodiment of this application. This method can be applied to one or more electronic devices such as smartphones, laptops, desktop computers, portable computers, and servers. Furthermore, the execution entity of this method can be hardware or software. When the execution entity is hardware, it can be one or more of the aforementioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the execution entity is software, this method can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are imposed here.

[0236] like Figure 3 As shown, the method specifically includes:

[0237] Step 301: Use a preset method to determine the severity of mosaic in the streaming media image, wherein the preset method is any of the mosaic severity detection methods described above.

[0238] In this embodiment, the specific implementation of step 301 is described above and will not be repeated here.

[0239] Step 202: Determine the bitrate of the streaming media image based on the severity of the mosaic.

[0240] In this embodiment, the bitrate of the streaming media image can be determined based on the severity of the mosaic using a pre-defined correspondence table, formula, or pre-fitted curve.

[0241] In some cases, the severity of pixelation can be negatively correlated with the bitrate of the streaming video. That is, the more severe the pixelation, the lower the bitrate of the streaming video.

[0242] Step 203: Transmit the outgoing data of the streaming media image according to the bitrate.

[0243] In this embodiment, after determining the bitrate of the streaming media image, the outgoing data of the streaming media image can be transmitted according to the bitrate.

[0244] Optionally, after transmitting the outgoing streaming data of the streaming media image according to the bitrate, the streaming media image can also be displayed.

[0245] In some optional implementations of this embodiment, after determining the severity of the mosaic effect in the streaming media image using a preset method, the following steps may also be performed:

[0246] The first step is to determine whether the severity of the mosaic is greater than or equal to a preset severity threshold.

[0247] The second step is to generate a prompt message indicating the severity of the mosaic in the streaming media image if the mosaic severity is greater than or equal to the preset threshold.

[0248] Optionally, after generating the above prompt information, the prompt information can be displayed, for example, during the display of a streaming media video.

[0249] It is understood that, among the above optional implementation methods, when the pixelation of the streaming media image is severe, a prompt message indicating the severity of the pixelation can be generated in a timely manner, so that users and other parties can understand the image quality of the streaming media image, thereby improving the user experience.

[0250] It should be noted that, in addition to the contents described above, this embodiment may also include... Figure 1 , Figure 2 The corresponding technical features described in the corresponding embodiments, thereby achieving Figure 1 , Figure 2 The technical effectiveness of the mosaic severity detection method shown above is described in detail in the relevant descriptions above. For the sake of brevity, it will not be elaborated here.

[0251] The image processing method provided in this application embodiment can use a preset method to determine the mosaic severity of a streaming media image. The preset method is the mosaic severity detection method described in the first aspect above. Then, based on the mosaic severity, the bitrate of the streaming media image is determined, and the outgoing data of the streaming media image is transmitted according to the bitrate. Therefore, different bitrates for the streaming media image can be determined based on different mosaic severity levels, thereby improving the matching degree between the transmission bitrate of the streaming media image and its mosaic severity.

[0252] The embodiments of this application are described below by way of example. However, it should be noted that the embodiments of this application may have the features described below, but the following description does not constitute a limitation on the protection scope of the embodiments of this application.

[0253] During the transmission of real-time streaming video (i.e., the aforementioned streaming media video), due to factors such as image complexity, network bandwidth, and encoding / decoding algorithms, a mosaic effect may occur. This effect manifests as randomly distributed, square-shaped pixel gaps in the video (i.e., the aforementioned streaming media video). The mosaic effect severely impacts image quality and observability; therefore, a method is needed to detect the severity of mosaic in real-time streaming video.

[0254] In video encoding, video quality is closely related to the encoding bitrate. The current HEVC (High Efficiency Video Coding) standard's R-λ model allocates the target bitrate to three levels: GOP (Group of Picture) level, image level, and basic unit level. The bitrate is allocated to each basic unit level, and then its encoding quantization parameter (QP) is calculated to make the output bitrate closer to the user-defined bitrate. While the model has achieved good results in achieving reasonable bitrate allocation and bitrate accuracy, it still has shortcomings. When allocating the bitrate, it does not consider the complexity of the video content itself, nor the spatiotemporal correlation of the video. Under the same network bandwidth, encoding QP, and transmission bitrate conditions, the more complex the scene (spatial complexity / temporal complexity), the higher the performance requirements for the codec. Details and high-frequency information (such as complex textures and edges) contained in complex scenes are more likely to be lost during compression; motion estimation in complex scenes is more difficult to predict accurately, leading to increased prediction errors; these lost information and error codes manifest as mosaic in the encoded video. In security scenarios, cameras are often installed outdoors for monitoring, where scenes are complex with many vehicles and pedestrians, resulting in a high frequency of mosaic in real-time streaming, severely impacting image quality and user satisfaction.

[0255] This scheme uses the spatial complexity of the image (i.e., the aforementioned streaming media image) and the motion intensity value of the image (i.e., the aforementioned time complexity) to evaluate the image complexity. By establishing a three-dimensional relationship between spatial complexity, motion intensity value, and mosaic severity, the corresponding SATD (Sum of Absolute Transformed Differences) value detected in the real-time streaming image is matched with the motion intensity value to determine the current mosaic severity, and real-time feedback processing is performed.

[0256] This solution requires the encoder to calculate the spatial complexity of the image in real time for bitrate allocation, and to calculate the spatial and temporal complexity of the image for strategy analysis, such as bitrate control, expectation management, and pop-up prompts. Utilizing the encoder's internal parameters, without adding additional algorithms, it can establish a matching relationship between SATD value (i.e., the aforementioned spatial complexity), motion vector value (i.e., the aforementioned time complexity), and mosaic severity (i.e., the aforementioned mosaic severity) to match the mosaic severity of the current image in real time. This information is then fed back to the encoder for bitrate control or to prompt the user for expectation management.

[0257] Specifically, please see Figure 4 ,Figure 4 This is a flowchart illustrating another image processing method provided in an embodiment of this application. The execution process of this solution includes:

[0258] 1. Switch between static scenes of different complexities, and fit a linear relationship between the spatial complexity value of the image and the degree of pixelation by obtaining the spatial complexity value of the image output by the encoder.

[0259] In the bitrate control module, the encoder can calculate the full frame complexity required for the bitrate control to select the best prediction mode during the encoding decision process, or to decide whether to perform block segmentation, and provide an appropriate quantization parameter QP value.

[0260] The QP value mentioned above can be used to control the degree of quantization. The formula for calculating the QP value is as follows:

[0261]

[0262] Where n represents the current frame number, starting from 0. i takes the value [0, n], used to identify the frames between frame 0 and frame n. When i = 0, ... The value is 1. R represents the target bit rate (number of bits), X is the current frame complexity (i.e., the space complexity mentioned above), and α is the code control parameter. SATD represents the sum of the residuals after Hadamard transformation, which can reflect a certain bitstream change. The SATDcost value in the frame-level QP calculation formula is the sum of the residuals STAD at half-pixel precision of the image, which can be calculated in the preprocessing module before the bit rate control module. Bitrate represents the amount of data per unit time, usually expressed in bps (bits per second). qscale represents the quantization level parameter. rceq is determined by the ratio of the cumulative value of all target bits up to the current encoded frame to the current frame complexity, reflecting the accuracy of the original quantization parameters calculated based on fuzzy complexity so far. fps (Frames Per Second) represents the number of frames captured / displayed per second. comp represents the non-linear mapping of SATD considering the characteristics of human vision, with a default value of 0.6. A value of 0 indicates CBR, and a value of 1 indicates fixed QP encoding. bit is an abbreviation for "Binary Digit", which is the smallest unit of computer data.

[0263] The complexity of the current frame is characterized by the SATD value. SATD is a metric for measuring residuals. It first calculates the difference (residual) between the current block (i.e., the frame block) and the reference block (i.e., the first reference block), then applies a Hadamard transform to these residuals, and finally calculates the sum of the absolute values ​​of the transformed residuals. Based on the SATD value output by the encoder, a near-linear relationship is fitted with the severity of mosaic in the subjective frame.

[0264] 2. By changing the motion level of the image in scenes with different levels of complexity, and by obtaining the motion vector values ​​output by the encoder, a linear relationship is fitted between the motion intensity value (time complexity value) and the pixelation level of the image.

[0265] The encoder can compute motion vector predictions. The HEVC standard proposes Merge mode and AMVP (Advanced Motion Vector Prediction) mode. In Merge mode, a candidate motion vector set (i.e., the motion vector set) can be built for the current PU using the motion information of the encoded PU (Prediction Unit, i.e., the aforementioned frame block). This set can contain 5 candidate motion vectors. The set is iterated through, and the one with the lowest rate-distortion cost is directly used as the current PU's MV (Motion Vector), replacing the motion estimation process for the current PU. The basic principle of Advanced Motion Vector Prediction (AMVP) is to build a candidate prediction vector list for the currently searched prediction unit PU using the motion vectors of its temporal and spatial neighboring blocks. In both modes, the encoder performs statistical optimization of motion vectors, using the motion vector values ​​to characterize the motion intensity value (i.e., time complexity) of the current frame. Based on the motion vector values ​​output by the encoder, a near-linear relationship is fitted with the mosaic severity of subjective frames of different complexities. For an example, please refer to [link to example]. Figure 5 , Figure 5 This diagram illustrates the relationship between the time complexity and the severity of mosaic detection methods provided in this application.

[0266] 3. Establish a built-in information table, which is a table showing the severity of mosaic corresponding to different bitrates, different image complexities, and different motion complexities.

[0267] 4. When the real-time streaming video is output, the SATD value and motion vector value output by the encoder are read in real time through the interface. When the value read under the current bitrate condition matches the mosaic level greater than the severity level, the encoding bitrate level is reduced and a prompt serial port reminder is displayed to remind the user that the current video mosaic is severe.

[0268] It should be noted that, in addition to the contents described above, this embodiment may also include the technical features described in the above embodiments, thereby achieving the technical effect of the mosaic severity detection method shown above. Please refer to the above description for details. For the sake of brevity, it will not be elaborated here.

[0269] This solution can utilize the parameters inside the encoder to read the SATD value and motion vector value of time complexity used by the encoder through an interface, without the need for additional algorithm detection. It can also pre-build the function of subjectively confirming the severity of mosaic, and finally realize the real-time detection of the severity of mosaic in the real-time stream, which can be used to adjust the bitrate level or prompt the user about the effect of the screen.

[0270] Figure 6 This is a schematic diagram of a mosaic severity detection device provided in an embodiment of this application.

[0271] Specifically, it includes:

[0272] The acquisition unit 401 is used to acquire outgoing data of the streaming media image captured in real time by the security camera, wherein the outgoing data is image data representing the streaming media image during transmission;

[0273] The first determining unit 402 is used to determine the picture complexity of the streaming media picture based on the outgoing data, wherein the picture complexity represents the degree of complexity of the streaming media picture;

[0274] The second determining unit 403 is used to determine the severity of mosaic in the streaming media image based on the image complexity.

[0275] In one possible implementation, the image complexity includes the spatial complexity of the streaming media image, which comprises at least two image blocks, and the spatial complexity represents the degree of difference between the image blocks of the streaming media image; and

[0276] Determining the video complexity of the streaming media image based on the outgoing data includes:

[0277] Based on the outflow data, determine the spatial complexity of the streaming media image; and

[0278] Determining the severity of mosaic artifacts in the streaming media image based on the image complexity includes:

[0279] Based on the aforementioned spatial complexity, the severity of the mosaic effect in the streaming media image is determined.

[0280] In one possible implementation, determining the spatial complexity of the streaming media image based on the outgoing data includes:

[0281] Based on the outgoing data, the streaming media image is divided into a first set of image blocks;

[0282] For a first frame block in the first frame block set, determine the residual between the first frame block and a first reference block, wherein the first reference block is the first frame block located in the streaming media frame;

[0283] Based on the determined residuals, the spatial complexity of the streaming media image is determined.

[0284] In one possible implementation, the image complexity includes the time complexity of the streaming media image, whereby the time complexity represents the degree of difference between historical images of the streaming media image and the current streaming media image; and

[0285] Determining the video complexity of the streaming media image based on the outgoing data includes:

[0286] Based on the outflow data, determine the time complexity of the streaming media frame; and

[0287] Determining the severity of mosaic artifacts in the streaming media image based on the image complexity includes:

[0288] Based on the time complexity, the severity of the mosaic effect in the streaming media image is determined.

[0289] In one possible implementation, determining the time complexity of the streaming media frame based on the outgoing data includes:

[0290] Based on the outgoing data, the streaming media image is divided into a second set of image blocks;

[0291] For the second frame block in the second frame block set, the motion vector of the second frame block is determined based on the second reference block, wherein the second reference block is the second frame block located in the historical frame;

[0292] The time complexity of the streaming media image is determined based on the determined motion vector.

[0293] In one possible implementation, the second reference block is determined in the following manner:

[0294] Determine the set of third screen blocks obtained by dividing the historical screens;

[0295] From the set of third frame blocks, determine the third frame blocks that are temporally or spatially adjacent to the second frame blocks in the streaming media frame to obtain a subset of frame blocks;

[0296] Determine the motion vectors of each third frame block in the frame block subset to obtain a motion vector set, wherein each third frame block in the frame block subset corresponds one-to-one with a motion vector in the motion vector set;

[0297] Determine the rate-distortion cost of the motion vectors in the set of motion vectors to obtain a rate-distortion cost set, wherein the rate-distortion cost in the rate-distortion cost set corresponds one-to-one with the third frame block in the frame block subset;

[0298] The frame block corresponding to the smallest rate-distortion cost in the rate-distortion cost set is determined as the second reference block.

[0299] In one possible implementation, determining the picture complexity of the streaming media image based on the outgoing data includes:

[0300] A first algorithm is used to determine the first complexity of the streaming media image based on the outgoing data, wherein the first algorithm is used to determine the complexity of the streaming media image based on the outgoing data, and the first complexity represents the complexity of the streaming media image determined by the first algorithm.

[0301] A second algorithm is used to determine the second complexity of the streaming media image based on the outgoing data. The second algorithm is used to determine the complexity of the streaming media image based on the outgoing data. The second algorithm is different from the first algorithm. The second complexity represents the complexity of the streaming media image determined by the second algorithm.

[0302] The image complexity of the streaming media image is determined based on the first complexity and the second complexity.

[0303] The mosaic severity detection device provided in this embodiment can be as follows: Figure 6 The mosaic severity detection device shown can perform all the steps of the mosaic severity detection methods described above, thereby achieving the technical effects of the mosaic severity detection methods described above. For details, please refer to the relevant descriptions above. For the sake of brevity, it will not be elaborated here.

[0304] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Specifically, it includes:

[0305] The third determining unit 411 is used to determine the severity of mosaic in the streaming media image using a preset device, wherein the preset device is the mosaic severity detection method described in the first aspect above.

[0306] The fourth determining unit 412 is used to determine the bitrate of the streaming media image based on the severity of the mosaic.

[0307] The transmission unit 413 is used to transmit the outgoing data of the streaming media image according to the bit rate.

[0308] In one possible implementation, after determining the severity of pixelation in the streaming media image using a preset device, the device further includes:

[0309] The fifth determining unit (not shown in the figure) is used to determine whether the severity of the mosaic is greater than or equal to a preset severity threshold.

[0310] The generation unit (not shown in the figure) is used to generate a prompt message indicating the severity of mosaic in the streaming media image when the mosaic severity is greater than or equal to the preset severity threshold.

[0311] The image processing device provided in this embodiment can be as follows: Figure 7 The image processing apparatus shown can execute all the steps of the image processing methods described above, thereby achieving the technical effects of the image processing methods described above. For details, please refer to the relevant descriptions above. For the sake of brevity, further details are not provided here.

[0312] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 The illustrated electronic device 500 includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 8 The general designated all buses as Bus System 505.

[0313] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0314] It is understood that the memory 502 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0315] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0316] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in application program 5022.

[0317] In this embodiment, by calling the program or instructions stored in memory 502, specifically the program or instructions stored in application program 5022, processor 501 executes the method steps provided in each method embodiment, including, for example:

[0318] Acquire outgoing data of streaming media images captured in real time by security cameras, wherein the outgoing data is image data representing the streaming media images during transmission;

[0319] Based on the outflow data, the image complexity of the streaming media image is determined, wherein the image complexity represents the degree of complexity of the streaming media image;

[0320] Based on the complexity of the image, the severity of the mosaic effect in the streaming media image is determined.

[0321] or,

[0322] A preset method is used to determine the severity of mosaic in the streaming media image, wherein the preset method is any of the mosaic severity detection methods described above;

[0323] The bitrate of the streaming media image is determined based on the severity of the mosaic.

[0324] The streaming data of the media image is transmitted according to the bitrate.

[0325] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0326] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described above in this application, or combinations thereof.

[0327] For software implementation, the techniques described herein can be implemented by units that perform the functions described above. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or external to the processor.

[0328] The electronic device provided in this embodiment may be as follows: Figure 8 The electronic device shown can perform all the steps of the detection methods for each degree of mosaic severity described above, thereby achieving the technical effects of the detection methods for each degree of mosaic severity described above. For details, please refer to the relevant descriptions above. For the sake of brevity, further details are not provided here.

[0329] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0330] When one or more programs in the storage medium can be executed by one or more processors, the above-described method for detecting the severity of mosaic artifacts executed on the electronic device side can be implemented.

[0331] The processor described above is used to execute a mosaic severity detection program stored in memory to implement the following steps of a mosaic severity detection method executed on the electronic device side:

[0332] Acquire outgoing data of streaming media images captured in real time by security cameras, wherein the outgoing data is image data representing the streaming media images during transmission;

[0333] Based on the outflow data, the image complexity of the streaming media image is determined, wherein the image complexity represents the degree of complexity of the streaming media image;

[0334] Based on the complexity of the image, the severity of the mosaic effect in the streaming media image is determined.

[0335] or,

[0336] A preset method is used to determine the severity of mosaic in the streaming media image, wherein the preset method is any of the mosaic severity detection methods described above;

[0337] The bitrate of the streaming media image is determined based on the severity of the mosaic.

[0338] The streaming data of the media image is transmitted according to the bitrate.

[0339] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0340] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0341] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0342] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method of detecting the severity of a mosaic, characterized in that, The method comprises: obtaining outflow data of a streaming media picture collected in real time by a security camera, wherein the outflow data is picture data representing the streaming media picture in a transmission process; determining picture complexity of the streaming media picture based on the outflow data, wherein the picture complexity represents a complexity degree of the streaming media picture; determining mosaic severity of the streaming media picture based on the picture complexity.

2. The method of claim 1, wherein, The picture complexity comprises spatial complexity of the streaming media picture, the streaming media picture comprises at least two picture blocks, and the spatial complexity represents a difference degree between the picture blocks of the streaming media picture. The determining of the picture complexity of the streaming media picture based on the outflow data comprises: determining the spatial complexity of the streaming media picture based on the outflow data; and The determining of the mosaic severity of the streaming media picture based on the picture complexity comprises: determining the mosaic severity of the streaming media picture based on the spatial complexity. The determining of the spatial complexity of the streaming media picture based on the outflow data comprises:

3. The method of claim 2, wherein, dividing the streaming media picture into a first picture block set based on the outflow data; determining a residual error between a first picture block in the first picture block set and a first reference block located in the streaming media picture; and determining the spatial complexity of the streaming media picture based on the determined residual error. The picture complexity comprises temporal complexity of the streaming media picture, and the temporal complexity represents a difference degree between a historical picture and the streaming media picture.

4. The method of claim 1, wherein, The determining of the picture complexity of the streaming media picture based on the outflow data comprises: determining the temporal complexity of the streaming media picture based on the outflow data; and The determining of the mosaic severity of the streaming media picture based on the picture complexity comprises: determining the mosaic severity of the streaming media picture based on the temporal complexity. The determining of the temporal complexity of the streaming media picture based on the outflow data comprises: dividing the streaming media picture into a second picture block set based on the outflow data; 5. The method of claim 4, wherein, determining a motion vector of a second picture block in the second picture block set based on a second reference block located in the historical picture; and determining the temporal complexity of the streaming media picture based on the determined motion vector. The second reference block is determined in the following manner: determining a third picture block set obtained by dividing the historical picture; 6. The method of claim 5, wherein, determining a third picture block adjacent to the second picture block in time or space in the streaming media picture from the third picture block set to obtain a picture block subset; determining a motion vector of each third picture block in the picture block subset to obtain a motion vector set, wherein the third picture blocks in the picture block subset and the motion vectors in the motion vector set are in one-to-one correspondence; and determining the mosaic severity of the streaming media picture based on the motion vector set. ​ determine rate-distortion costs of the motion vectors in the set of motion vectors to obtain a set of rate-distortion costs, wherein a rate-distortion cost in the set of rate-distortion costs corresponds to a third picture block in the subset of picture blocks one-to-one; determine, as the second reference block, the third picture block corresponding to the rate-distortion cost with the smallest value in the set of rate-distortion costs.

7. The method of claim 1, wherein, The determining, based on the outflow data, of the picture complexity of the streaming media picture comprises: determining, based on the outflow data, a first complexity of the streaming media picture by using a first algorithm, wherein the first algorithm is used to determine the complexity of the streaming media picture based on the outflow data, and the first complexity represents the complexity of the streaming media picture determined by using the first algorithm; determining, based on the outflow data, a second complexity of the streaming media picture by using a second algorithm, wherein the second algorithm is used to determine the complexity of the streaming media picture based on the outflow data, the second algorithm is different from the first algorithm, and the second complexity represents the complexity of the streaming media picture determined by using the second algorithm; determining the picture complexity of the streaming media picture based on the first complexity and the second complexity.

8. An image processing method characterized by, The method comprises: determining the mosaic severity of the streaming media picture by using a preset method, wherein the preset method is the detection method of the mosaic severity according to any one of the preceding claims 1-7; determining the code rate of the streaming media picture based on the mosaic severity; transmitting the outflow data of the streaming media picture according to the code rate.

9. The method of claim 8, wherein, After the determining of the mosaic severity of the streaming media picture by using the preset method, the method further comprises: determining whether the mosaic severity is greater than or equal to a preset degree threshold; generating prompt information indicating the mosaic severity of the streaming media picture in the case that the mosaic severity is greater than or equal to the preset degree threshold.

10. A device for detecting the severity of a mosaic, characterized in that The device comprises: an acquisition unit configured to acquire outflow data of a streaming media picture collected by a security camera in real time, wherein the outflow data is picture data representing the streaming media picture in a transmission process; a first determination unit configured to determine a picture complexity of the streaming media picture based on the outflow data, wherein the picture complexity represents the complexity of the streaming media picture; a second determination unit configured to determine a mosaic severity of the streaming media picture based on the picture complexity.

11. An image processing apparatus characterized by comprising: The device comprises: a third determination unit configured to determine the mosaic severity of the streaming media picture by using a preset device, wherein the preset device is the detection method of the mosaic severity according to any one of the preceding claims 1-7; a fourth determination unit configured to determine the code rate of the streaming media picture based on the mosaic severity; a transmission unit configured to transmit the outflow data of the streaming media picture according to the code rate.

Images