A real-time object stationary detection method and system based on image fuzziness

Abstract

The present invention discloses a real-time object stillness detection method and system based on image blur, including converting the current frame color image in the video stream into a grayscale image and updating the grayscale image sequence of fixed length in real time; calculating the blur of the grayscale image and updating the blur sequence in real time; using the incremental update method to update the average image in real time and calculate the blur of the average image; taking the ratio of the blur of the average image to the maximum blur in the blur sequence as the relative stillness coefficient, and comparing it with the set threshold to judge the motion state of the scene. The present invention obtains the relative stillness coefficient by fusing continuous multi-frame information, which can be real-time, accurate, and has good stability, and can effectively resist the interference of noise and illumination changes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time object stillness detection method and system based on image fuzziness. Background Art

[0002] In recent years, under the wave of digitalization and intelligence, the catering industry is experiencing a drastic change. Among them, dish recognition technology, as the pioneer of this change, is gradually changing consumers' dining experience. Unlike traditional identification technologies such as RFID, which have a certain loss rate, the latest generation of dish recognition technology only relies on cameras to collect and identify dishes, achieving truly senseless and zero-loss recognition. The key to this technology is the use of intelligent visual recognition algorithms. The process is as follows: consumers select dishes and place them on the smart dining table, the camera collects dish images and transmits them to the background server, and the server uses deep learning and other technologies to detect, feature analyze and identify dish images to determine the type and price of the dishes.

[0003] In the above-mentioned dish recognition process, it is particularly important to detect the static moment of the dish. If the image is captured when the dish is not completely static, the captured image will be severely blurred, which will greatly affect the accuracy of subsequent detection, feature extraction and recognition algorithms, and may even make the visual recognition algorithm invalid, resulting in incorrect dish types and prices. Currently, common methods in the field of static dish detection include frame difference method and optical flow method.

[0004] The frame difference method calculates the difference between two adjacent frames to determine whether the image is still. Therefore, the frame difference method is easily affected by changes in illumination, especially when there is small motion or noise interference in the image. The frame difference method may not be able to accurately determine the still state. In addition, the frame difference method mainly compares adjacent frames and lacks the use of the overall information of the time series, which may lead to instability in judgment.

[0005] The optical flow method detects motion by calculating the motion vectors of pixels between consecutive frames. It can obtain rich motion information and is also highly robust to noise and lighting changes. However, the computational complexity is high and it is not suitable for scenarios that require high real-time performance, such as canteen dish recognition with sudden large traffic. On the other hand, the still detection method based on deep learning uses a trained model to judge dish images. This type of method is more robust in complex scenarios, but requires a large amount of labeled data sets for training, and its high computational complexity also makes it difficult to meet high real-time demand scenarios. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a real-time object stillness detection method based on image blur, which uses the blurriness of continuous frame images to detect stillness of objects for the first time. The method comprises the following steps:

[0007] S1, initialize the cached fixed-length image sequence, convert each color image in the sequence into a grayscale image, calculate the blur of each image in the image sequence, and obtain an initialized blur sequence;

[0008] S2, converting the current frame image into a grayscale image, and calculating the blurriness of the current frame image;

[0009] S3, update the image sequence by using the in-out-out operation and update the blur sequence;

[0010] S4, using the incremental update method to calculate the average image of the updated image sequence, and calculating the blurriness of the average image;

[0011] S5. Use the traversal method to obtain the maximum blur in the blur sequence, calculate the ratio of the blur of the current average image and the maximum blur of the sequence, and use it as the relative stillness coefficient. If the relative stillness coefficient is greater than the preset threshold, it means that the scene is in a still state at the current moment, otherwise it is in a moving state.

[0012] Furthermore, the S1 specifically includes the following steps: using the Laplace operator to calculate the Laplace feature image, calculating the variance according to the obtained Laplace image features, and using the calculated variance as the blur of the grayscale image.

[0013] Further, the using the Laplacian operator to calculate the Laplacian eigenimage comprises:

[0014] Use the Laplacian operator to calculate the grayscale changes around each pixel in the image to obtain edge information, and discretize the Laplacian operator:

[0015]

[0016] Grayscale image at frame t Convolution operation is performed on , and get the Laplace eigenimage .

[0017] Further, calculating the variance according to the obtained Laplace image features includes:

[0018]

[0019] in Represents the value of the kth pixel in the Laplace feature image, Represents the mean value of the Laplace feature.

[0020] Furthermore, the updating of the image sequence and the blur sequence using the incremental updating method specifically includes:

[0021] Pop the first element in the sequence and save it in a temporary variable, and insert the new element at the end of the sequence.

[0022] Furthermore, the incremental update method is used to calculate the average image, and then the blur is calculated from the average image. The average image is only at time When N is the length of the given image and blur sequence, it is defined as follows:

[0023] When the moment When the sum of the sequence images is initialized ;

[0024] When the moment When ;

[0025] When the moment When , calculate the average image , calculate the average image blur ;

[0026] According to another aspect of the specification, a real-time object stillness detection system based on image fuzziness for implementing the method is also provided, the system comprising: an image sequence preprocessing module, a fuzziness calculation module, an incremental update module and an object stillness detection module;

[0027] The image sequence preprocessing module is used to obtain the current frame and convert it into a grayscale image;

[0028] The fuzziness calculation module is used to calculate the fuzziness of the grayscale image and the average image;

[0029] The incremental update module updates the image sequence and the blur sequence using the in-column and out-column operation, and calculates the average image using the incremental update method;

[0030] The object stillness detection module calculates the ratio of the blurriness of the average image at the current moment to the maximum blurriness of the sequence, and defines it as a relative stillness coefficient. If the relative stillness coefficient is greater than a preset threshold, the scene is still at the current moment, otherwise it indicates that there is motion in the scene.

[0031] According to another aspect of the specification, a real-time object stillness detection device based on image blur is also provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the real-time object stillness detection method based on image blur is implemented.

[0032] According to another aspect of the specification, a computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the real-time object stillness detection method based on image blur is implemented.

[0033] According to another aspect of the specification, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the real-time object stillness detection method based on image blur is implemented.

[0034] Beneficial effects of the present invention:

[0035] (1) This method uses the ratio of the average image blur to the maximum blur of the sequence to obtain the relative stillness coefficient, and uses the relative stillness coefficient to determine whether the image is still. This method combines the blur information of multiple frames to obtain the relative stillness coefficient, which can more comprehensively reflect the motion state of the object and has strong resistance to noise. The calculation of the average blur can integrate the information in the time series and provide a more stable basis for judgment.

[0036] (2) Compared with the optical flow method, this algorithm has a significant advantage in judgment speed by incrementally updating the average image. This incremental update mechanism only requires simple addition and subtraction operations on the data of the new frame and the old frame, which significantly reduces the amount of calculation, especially when processing high frame rate videos, which can greatly improve the real-time processing capability. In addition, this method can respond quickly and is suitable for real-time analysis application scenarios, such as real-time scenes of dish recognition. At the same time, it has low requirements for CPU and memory resources and is suitable for running in resource-constrained environments. In terms of implementation, the incremental update algorithm is relatively simple, easy to integrate and debug, and reduces the complexity of development and maintenance. In contrast, the optical flow method involves complex motion vector calculations in the calculation process, which not only increases the amount of calculation, but also leads to slower processing speed and high computing resource requirements.

[0037] (3) Compared with the static detection method based on deep learning, this method has several significant advantages. First, the computational efficiency is higher. The fuzzy calculation relies on simple image processing operations, which is fast and requires few resources, making it suitable for real-time applications. Second, the method is simple to implement, does not require large-scale datasets and complex training processes, and is easy to understand and deploy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the structure of a still detection method based on image blur provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of calculating the static coefficient provided by an embodiment of the present invention;

[0040] Figure 3A schematic diagram of incremental update of the present invention provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of average image calculation under different conditions provided by an embodiment of the present invention;

[0042] Figure 5 A comparison diagram of static judgment of brightness influence of the patented method provided in an embodiment of the present invention;

[0043] Figure 6 The frame difference method provided in the embodiment of the present invention is used to judge the static contrast under the influence of brightness;

[0044] Figure 7 A schematic diagram of a real-time object stillness detection device based on image blur provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0046] like Figure 1 As shown, in order to solve the problem of stillness detection of moving objects, the present invention proposes a method for stillness detection of objects based on image blur, and the specific steps are as follows:

[0047] (1) Initialize the cache of N frames of images to obtain an image sequence. For example, N = 10, which means storing the most recent 10 frames of images. Convert the N frames of images into grayscale images for subsequent blur calculation and incremental update.

[0048] (2) For the image sequence obtained in step (1), the blur of each frame in the sequence is calculated to obtain a blur sequence. Blur is an important indicator for measuring image clarity. By evaluating the blur of an image, it can be determined whether the image is clear. The blur calculation method comprises the following steps:

[0049] (2-1) The Laplacian operator is a second-order derivative operator that can detect edges and details in an image. The basic principle is to obtain edge information by calculating the grayscale changes around each pixel in the image. The Laplacian feature image of a blurred image usually shows fewer edges and flatter areas. The discrete Laplacian operator h:

[0050]

[0051] For grayscale images Convolution operation is performed on , and get the Laplace eigenimage .

[0052] (2-2) Calculate the variance based on the Laplace image features obtained in (2-1) As the blurriness of a grayscale image, the larger the variance, the wider the response of the image, and the clearer the image; the smaller the variance, the smaller the response range, and the blurrier the image.

[0053] (3) Obtain the current frame image, convert the current frame image into a grayscale image, and calculate the blurriness of the current frame image. The blurriness calculation method is the same as steps (2-1) to (2-2) in step (2).

[0054] (4) For the image sequence obtained in step (1), the ambiguity sequence obtained in step (2), and the current frame image and current frame ambiguity obtained in step (3), the image sequence and the ambiguity sequence are enqueued and dequeued. The specific method includes the following steps:

[0055] (4-1) Read the N-frame image sequence and blur sequence of the previous moment.

[0056] (4-2) Update the image sequence in step (1) with the current frame image in step (3). The image sequence update includes popping up the starting frame in the sequence , the current frame image Insert at the end of the sequence.

[0057] (4-3) Update the ambiguity sequence in step (2) with the current frame ambiguity in step (3). The ambiguity sequence update includes the starting frame ambiguity in the pop-up sequence , the current frame blur Insert at the end of the sequence.

[0058] (5) The incremental update method is used to calculate the average image and the average image blur. Only the necessary image data is updated, which avoids the repeated summation of the entire image sequence each time the average image is calculated. This improves the processing speed and efficiency and is suitable for real-time applications. It should be noted that the average image is only updated at time It is defined as, where N is the length of the given image and blur sequence, when the moment When the sum of the sequence images is initialized ; When the moment When , the cumulative length of the image sequence has not yet reached the given sequence length N, so the average image cannot be completely calculated. ; When the moment When , calculate the average image , the specific method at this time includes the following steps:

[0059] (5-1) Read the sum of the N frames of the previous image sequence , the sum of N frame sequence images is

[0060] (5-2) Update the sum of the images. First, subtract the starting frame image of the sequence from the accumulated sum at the previous moment. , that is: then add the current frame image , the two operation steps can be unified into the following expression: .

[0061] in, is the sum of N frames of images at the current moment, is the sum of N frames of images at the previous moment, is the current frame image, is the first frame in the sequence of images. The specific implementation process is as follows Figure 3 shown.

[0062] (5-3) Calculate the average image , calculate the blur of the average image , the blur calculation method is the same as steps (2-1) to (2-2) in step (2). The basis for using the blur of the average image as the basis for judging whether the object or scene is still is: in a still state, each frame of the image captured contains the same static scene, the image content between different frames is completely aligned, and the final generated average image is clear and rich in details. However, in the case of object or camera movement, the image content between different frames is not completely aligned. When the image sequence is averaged, blurring will occur at the edges and high-frequency textures of the moving object. Therefore, the blur of the average image in the moving state will be significantly higher than the blur in the still state, such as Figure 4 As shown, the average image of the moving scene (red outer border) is more easily blurred than the average image of the stationary scene (blue outer border).

[0063] (6) Calculate the maximum ambiguity in the updated ambiguity sequence obtained in step (4-3). The maximum ambiguity is .

[0064] (7) In the prior art, blur is usually determined by a fixed threshold to determine whether an image is clear or not. This can only reflect the clarity of the image but cannot independently determine whether the image is still or not. Although the average image blur of the sequence can be used as a reference for stillness judgment, it is difficult to set a unified and effective threshold standard for stillness judgment due to the drastic changes in the numerical range of the average image blur in different scenes. This method provides a more effective stillness judgment mechanism by calculating the ratio of the average image blur and the maximum blur of the sequence. It is suitable for capturing still moments in different scenes and changing lighting conditions. The specific method is as follows: Calculate the ratio between the maximum blur of the sequence obtained in step (6) and the average image blur obtained in step (5) to obtain a relative stillness coefficient R. The calculation formula for this coefficient is:

[0065]

[0066] Since the average image will not be clearer than the clearest image in the sequence, the relative stillness coefficient will not be greater than 1. When the ratio of the two is close to 1, it means that the blurriness of the average image is equivalent to the maximum blurriness of the sequence. It can be further obtained that the average image is close to the clearest image in the sequence. This situation can only occur when it is still or in violent motion. Since the frame rate of most commercial cameras is at least 30fps, it is almost impossible for each frame of the sequence to move evenly and violently. Only when it is still can the average image be close to the clearest image; when the ratio of the two is small, it means that the average image is very different from the clearest image in the sequence, indicating that the object has a high probability of moving within N frames. In actual application scenarios, experimental verification has found that setting the threshold V=0.8 for judging stillness can achieve better results. When the coefficient R>0.8, it is judged that the scene at the current moment is in a stationary state, and when the coefficient R<0.8, it is judged that the scene at the current moment is in a moving state. It is worth emphasizing that the relative stillness coefficient, as an absolute indicator, has a strong universality and is applicable to various indoor scenes, especially in the application of dish recognition. The design of this method enables it to maintain stable performance in a variety of restaurant environments. Regardless of frequent changes in lighting, complex dish placement, or the presence of dynamic objects in the scene (such as chefs or customers), the algorithm can effectively perform still detection, ensuring that the dish images used for identification are clear and improving the accuracy of dish recognition.

[0067] (8) Repeat steps (3) to (7).

[0068] Figure 5 The difference between the static sequence in the static constant-brightness image sequence and the static dimming image sequence is shown. It can be seen that the stability of this method is good and the designed static coefficient is less affected by the change of illumination. Figure 6 The difference in coefficients between the frame difference method in a static bright image sequence and a static darkened image sequence is given, showing that the frame difference method's judgment index changes dramatically under light interference, exposing its limitations in a changing environment. Table 1 shows the time loss of different methods for video calculation with a resolution of 1390*750 in the Windows Python running environment. The average single-frame processing time required by this method is significantly lower than that required by the optical flow method.

[0069] Table 1

[0070] method This patented method Optical flow method Frame Difference Method Single frame processing time (milliseconds / ms) 75.64 232.86 10.92

[0071] Corresponding to the above-mentioned embodiment of a real-time object stillness detection method based on image fuzziness, the present invention also provides an embodiment of a real-time object stillness detection system based on image fuzziness, the system comprising: an image sequence preprocessing module, a fuzziness calculation module, an incremental update module and an object stillness detection module;

[0072] The image sequence preprocessing module is used to obtain the image sequence of the previous moment and the current frame and convert them into grayscale images

[0073] The fuzziness calculation module is used to calculate the fuzziness of each grayscale image in the image sequence and the current frame to obtain the image fuzziness sequence and the fuzziness of the current frame;

[0074] The incremental update module updates the image sequence using the incremental update method, and after the update is completed, the fuzziness calculation module is used to calculate the average image of the updated image sequence and the fuzziness of the average image;

[0075] The object stillness detection module uses the ratio of the average image blur of the updated image sequence to the maximum blur of the sequence as the relative stillness coefficient. If the relative stillness coefficient is greater than a preset threshold, the content in the image is in a still state at the current moment, otherwise it is in a moving state at the current moment.

[0076] Corresponding to the aforementioned embodiment of a real-time stationary object detection method based on image blur, the present invention also provides an embodiment of a real-time stationary object detection device based on image blur.

[0077] See also Figure 7 A real-time object stillness detection device based on image blurriness provided by an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a real-time object stillness detection method based on image blurriness in the above embodiment.

[0078] An embodiment of a real-time object stillness detection device based on image blur provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 7 As shown in FIG. 1 , a hardware structure diagram of a device having data processing capability for real-time object stillness detection based on image fuzziness provided by the present invention is shown in FIG. Figure 7In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0079] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0080] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0081] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a real-time object stillness detection method based on image blur in the above embodiment is implemented.

[0082] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0083] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the real-time object stillness detection method based on image blur is implemented.

[0084] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0085] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A real-time object stationary detection method based on image fuzziness, characterized in that: The method comprises the following steps: S1, initialize the cached fixed-length image sequence, convert each color image in the sequence into a grayscale image, calculate the blur of each image in the image sequence, and obtain an initialized blur sequence; The S1 specifically comprises the following steps: using the Laplace operator to calculate the Laplace feature image, calculating the variance according to the obtained Laplace image features, and using the calculated variance as the blur of the grayscale image; The method of calculating the Laplacian eigenimage using the Laplacian operator comprises: Use the Laplacian operator to calculate the grayscale changes around each pixel in the image to obtain edge information, and discretize the Laplacian operator: In the tth frame grayscale image I t Perform convolution operation on it* to obtain the Laplace feature image L t =h*I t ; Calculating the variance according to the obtained Laplace image features comprises: Where L k Represents the value of the kth pixel in the Laplace feature image, represents the mean value of the Laplacian image; S2, converting the current frame image into a grayscale image, and calculating the blurriness of the current frame image; S3, update the image sequence by using the in-out-out operation and update the blur sequence; S4, using the incremental update method to calculate the average image of the updated image sequence, and calculating the blur of the average image; the use of the incremental update method to calculate the average image of the updated image sequence includes: the average image is only defined at time t≥N, where N is the length of the given image and blur sequence, specifically including: When time t = 0, the sum of the sequence images S is initialized t =0; When the time t < N, calculate S t = S t-1 + I t ; When time t≥N, calculate S t =S t-1 -I t-N +I t , calculate the average image Calculate the average image blur S5. Use the traversal method to obtain the maximum blur in the blur sequence, calculate the ratio of the blur of the current average image and the maximum blur of the sequence, and use it as the relative stillness coefficient. If the relative stillness coefficient is greater than the preset threshold, it means that the scene is in a still state at the current moment, otherwise it is in a moving state.

2. The real-time object stillness detection method based on image fuzziness according to claim 1, characterized in that: The method of updating the image sequence and the blur sequence using the input sequence specifically includes the following steps: popping out the first element in the sequence and saving it in a temporary variable, and inserting the newly added element into the end of the sequence.

3. A real-time object stillness detection system based on image blurriness for implementing the method according to any one of claims 1 to 2, characterized in that: The system comprises: an image sequence preprocessing module, a fuzziness calculation module, an incremental update module and an object stillness detection module; The image sequence preprocessing module is used to obtain the current frame and convert it into a grayscale image; The fuzziness calculation module is used to calculate the fuzziness of the grayscale image and the average image; The incremental update module updates the image sequence and the blur sequence using the in-column-out operation and calculates the average image using the incremental update method; The object stillness detection module calculates the ratio of the blurriness of the average image at the current moment to the maximum blurriness of the sequence, and defines it as a relative stillness coefficient; if the relative stillness coefficient is greater than a preset threshold, the scene is still at the current moment, otherwise it indicates that there is motion in the scene.

4. A real-time object stillness detection device based on image blur, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a real-time object stillness detection method based on image blur as described in any one of claims 1-2 is implemented.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a real-time object stillness detection method based on image blur as described in any one of claims 1 to 2 is implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, a real-time object stillness detection method based on image blur is implemented as described in any one of claims 1 to 2.

Images