Water adding detection method and device based on image recognition, equipment and medium

Abstract

The invention provides a water adding detection method, device and equipment based on image recognition and a medium. The problem that an existing humidifier cannot accurately recognize water adding through an image recognition technology on the basis of avoiding dynamic water mist interference is effectively solved. The method comprises the steps that an image collecting module collects a real-time image sequence above an air outlet of the humidifier in real time, and a dynamic water mist target image sequence in the real-time image sequence is removed through a control module; the control module recognizes a water adding action and obtains multiple pieces of position information of the target bounding box; the control module performs fitting based on the multiple pieces of position information to obtain a water adding movement track and a future position relation; and the control module fuses the target image sequence, the future position relation and the target distance between the target bounding box and the air outlet to obtain a water adding intention and generate a shutdown instruction, and controls the water adding execution mechanism to execute the shutdown instruction so as to carry out safe water adding.

Description

Technical Field

[0001] This application relates to the field of image recognition, and more specifically, to a method, apparatus, device, and medium for detecting water addition based on image recognition. Background Technology

[0002] Most existing humidifiers use a top-fill design with the air outlet facing upwards, allowing users to add water through the outlet. Specifically, current technology typically relies on a water level sensor to detect the water tank's status and uses indicator lights to prompt the user to add water through the outlet when needed. However, when the humidifier is operating (fan running), if the user adds water directly, the fan airflow may blow water droplets backwards, causing water to splash into the external environment or inside the device, affecting cleanliness, user experience, and even causing electrical malfunctions. The humidifier cannot anticipate the user's intention to add water, especially in the event of sudden water addition during operation, lacking an effective preventative mechanism.

[0003] Image recognition technology is widely used in smart home appliances, such as obstacle avoidance in robot vacuum cleaners. Traditional image processing methods are mostly for static target recognition, while the dynamic water mist generated by humidifiers during operation interferes with traditional image processing and cannot meet the real-time requirements of user behavior. However, existing technologies have not been used to predict the water-filling behavior of humidifier users and to achieve active shutdown.

[0004] Therefore, there is a need to provide an image recognition-based water addition detection method that can avoid interference from dynamic water mist. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a water addition detection method, device, equipment and medium based on image recognition, which effectively solves the problem that existing humidifiers cannot accurately identify water addition by using image recognition technology while avoiding dynamic water mist interference.

[0006] In a first aspect, embodiments of this application provide an image recognition-based water addition detection method, applicable to a water addition control system. The water addition control system includes an image acquisition module, a control module, and a water addition actuator. The method includes: The image acquisition module acquires real-time image sequences above the air outlet of the humidifier, obtains the image characteristics of the real-time image sequences through the control module, and preprocesses the real-time image sequences based on the image characteristics to obtain the target image sequence; The control module controls the pre-trained water-adding recognition model to identify the water-adding action based on the target bounding box in the target image sequence, and obtains multiple positional information of the target bounding box in the target image sequence; The control module fits the water-adding motion trajectory based on the multiple location information, and predicts the future positional relationship between the target bounding box and the air outlet within a future time period based on the water-adding motion trajectory. The control module integrates the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water filling intention. Based on the water filling intention, it generates a shutdown command and controls the water filling actuator to execute the shutdown command to perform safe water filling.

[0007] In conjunction with the first aspect, this application provides a first possible implementation of the first aspect, wherein the control module fuses the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water addition intention, including: The evaluation data corresponding to different combinations of water-pouring container, hand, and water flow in the target image within a consecutive frame number are statistically analyzed; wherein the water-pouring container, hand, and water flow also constitute a water-adding action; The intention to add water is determined based on the evaluation data corresponding to the different combinations, the numerical comparison results of the target distance and the corresponding adaptive threshold, and the future positional relationship.

[0008] In conjunction with the first aspect, this application provides a second possible implementation of the first aspect, wherein the adaptive threshold corresponding to the target distance is determined based on the following steps: The historical target images and corresponding historical target distances corresponding to the multiple historical water filling and shutdown mechanisms of the humidifier are obtained and calculated respectively to obtain the calculation results; Based on the calculation results, the initial threshold of the target distance is adjusted to obtain the adaptive threshold.

[0009] In conjunction with the first aspect, this application provides a third possible implementation of the first aspect, wherein, after determining the water-adding intention based on the evaluation data, the target distance and the corresponding adaptive threshold, and the future positional relationship, the following steps are included: Determine whether the evaluation data of the hand and the target distance meet the preset abnormal water filling conditions and generate the corresponding judgment result; Based on the judgment result, a water filling and shutdown mechanism is triggered to control the humidifier to stop responding to the water filling and shutdown mechanism.

[0010] In conjunction with the first aspect, this application provides a fourth possible implementation of the first aspect, wherein the preprocessing of the real-time image sequence based on the image features to obtain the target image sequence includes: Each frame of the real-time image in the real-time image sequence is processed according to the processing method corresponding to the image characteristics to obtain a binary image including dynamic water mist region and non-dynamic water mist region; The dynamic water mist region in the binary image corresponding to each real-time image frame is filtered to obtain the target image sequence composed of images corresponding to non-dynamic water mist regions.

[0011] In conjunction with the first aspect, this application provides a fifth possible implementation of the first aspect, wherein processing each frame of the real-time image in the real-time image sequence according to the processing method corresponding to the image characteristics to obtain a binary image including dynamic water mist regions and non-dynamic water mist regions includes: Frequency domain analysis is performed on each frame of the real-time image to distinguish the high-frequency and low-frequency regions in each frame of the real-time image; Based on the features of the dynamic water mist region, high-frequency regions in each frame of real-time image are extracted to distinguish between the dynamic water mist region and the non-dynamic water mist region.

[0012] In conjunction with the first aspect, this application provides a sixth possible implementation of the first aspect, wherein predicting the future positional relationship between the target bounding box and the air outlet within a future time period based on the water-adding trajectory includes: Based on the water-addition trajectory, the future height data of the target bounding box within a preset future time period can be predicted; The future height data is compared with the pre-acquired height data of the air outlet to determine the future positional relationship.

[0013] Secondly, embodiments of this application provide an image recognition-based water addition detection device, applicable to water addition control systems. The water addition control system includes an image acquisition module, a control module, and a water addition actuator. The device includes: The acquisition module is used to acquire real-time image sequences above the air outlet of the humidifier in real time. The control module obtains the image characteristics of the real-time image sequence and preprocesses the real-time image sequence based on the image characteristics to obtain the target image sequence. The control module is used to control the pre-trained water addition recognition model to identify the water addition action based on the target bounding box in the target image sequence, and to obtain multiple positional information of the target bounding box in the target image sequence. The fitting module is used by the control module to fit the water-adding motion trajectory based on the multiple location information, and to predict the future positional relationship between the target bounding box and the air outlet in the future time period based on the water-adding motion trajectory. The water filling module is used by the control module to fuse the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water filling intention, generate a stop command based on the water filling intention, and control the water filling actuator to execute the stop command to perform safe water filling.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of any of the image recognition-based water addition detection methods described above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of any one of the image recognition-based water addition detection methods.

[0016] This application provides an image recognition-based water addition detection method applicable to water addition control systems. The water addition control system includes an image acquisition module, a control module, and a water addition actuator. The method first uses the image acquisition module to acquire a real-time image sequence above the air outlet of the humidifier. The control module then obtains the image characteristics of the real-time image sequence and preprocesses it based on these characteristics to obtain a target image sequence. Next, the control module controls a pre-trained water addition recognition model to identify the water addition action based on the target bounding box in the target image sequence, and obtains multiple positional information of the target bounding box in the target image sequence. Then, the control module simulates the water addition action based on the multiple positional information. The system obtains the water-adding trajectory and predicts the future positional relationship between the target bounding box and the air outlet within a future time period based on the trajectory. Finally, the control module fuses the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water-adding intention. Based on the water-adding intention, a shutdown command is generated, and the water-adding actuator is controlled to execute the shutdown command for safe water addition. This not only ensures the accuracy of water addition identification based on image recognition technology but also avoids interference caused by dynamic water mist. Furthermore, it achieves the identification of the water-adding effect using image recognition technology while avoiding the influence of dynamic water mist, thereby controlling the water-adding actuator to shut down and ensuring the safety of personnel and equipment during water addition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of an image recognition-based water addition detection method provided in an embodiment of this application is shown. Figure 2 A schematic diagram of the process for obtaining image sequences provided in an embodiment of this application is shown; Figure 3 This application provides a schematic diagram of the process for obtaining the intention to add water, as illustrated in an embodiment of the present application. Figure 4 This paper shows a structural block diagram of a water addition detection device based on image recognition provided in an embodiment of this application; Figure 5 A structural block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0020] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Image recognition technology is widely used in smart home appliances, such as obstacle avoidance in robotic vacuum cleaners. However, when applied to humidifiers, it is mostly for recognizing static targets. It is difficult to avoid interference from dynamic water mist during humidifier operation and cannot meet the real-time requirements of user behavior. As a result, humidifiers cannot recognize the user's intention to add water in advance, especially when water is suddenly added during operation, and there is a lack of effective prevention mechanisms.

[0023] Based on this, this application provides a water addition detection method, apparatus, device, and medium based on image recognition, which will be described below through embodiments.

[0024] Example 1 To facilitate understanding of this embodiment, a water addition detection method based on image recognition disclosed in this application will first be described in detail. For example... Figure 1 The flowchart shown illustrates a water addition detection method based on image recognition. This application provides a water addition detection method based on image recognition, applicable to water addition control systems. The water addition control system includes an image acquisition module, a control module, and a water addition actuator. The method includes: S101, The image acquisition module acquires a real-time image sequence above the air outlet of the humidifier, obtains the image characteristics of the real-time image sequence through the control module, and preprocesses the real-time image sequence based on the image characteristics to obtain the target image sequence; S102, The control module controls the pre-trained water-adding recognition model to recognize the water-adding action based on the target bounding box in the target image sequence, and obtains multiple positional information of the target bounding box in the target image sequence; S103. The control module fits the water-adding motion trajectory based on the multiple location information, and predicts the future positional relationship between the target bounding box and the air outlet within a future time period based on the water-adding motion trajectory. S104. The control module integrates the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water filling intention. Based on the water filling intention, it generates a shutdown command and controls the water filling actuator to execute the shutdown command to perform safe water filling.

[0025] In this application, the water filling control system can be applied to household appliances that require water filling, such as humidifiers. This application uses a humidifier as an example for illustration. The water filling control system includes an image acquisition module, a control module, and a water filling actuator. The image acquisition module is a miniature CMOS camera installed at the edge of the air outlet, with the lens facing upwards and tilted at 30° to avoid direct spraying of water mist. The image acquisition module is also equipped with an infrared LED (850nm, 0.5W) to enable the image acquisition module to support low-light conditions. The control module has a built-in image processing unit and a water filling recognition model, which is implemented based on a 168MHz, 512KB memory STM32F407 microcontroller, running a pruned water filling recognition model. The water filling actuator includes a fan and an atomizing device, which can be in the form of ultrasonic or cold evaporation. Both enable the water in the humidifier to be diffused into the room in the form of water mist through the air outlet. If the user is operating the water filling actuator, it will affect the user's safety. Therefore, the operating status of the water filling actuator is controlled by the control module to ensure the user's safety. The water filling control system also includes a power module to provide power to the water filling control system.

[0026] In step S101, the image acquisition module acquires a real-time image sequence above the humidifier's air outlet at a preset resolution of 640×480 and a preset frame rate of 10fps. When a moving object is detected in a frame of the real-time image sequence, the frame rate of the image acquisition module is dynamically adjusted from 10fps to 15fps to capture fast-moving objects in the real-time image sequence. The resolution is set based on the processing capability and real-time processing requirements of the control module. After obtaining the real-time image sequence, the real-time image sequence is transmitted via a pre-established communication link between the image acquisition module and the control module. The data is transmitted to the control module, where a preprocessing module extracts the image features of each frame of the real-time image sequence. These features include the presence of dynamic water mist and noise. The dynamic water mist is generated by the air outlet of the humidifier during the operation of the water filling mechanism. A preprocessing method is pre-set for the dynamic water mist and noise to remove them from the real-time image sequence and obtain the target image sequence. The preprocessing method for the dynamic water mist is inter-frame difference removal, and the preprocessing method for the noise is Gaussian blur, specifically Gaussian blur (5x5 kernel, σ=1.0) to further smooth the noise in the image sequence.

[0027] In the specific implementation of step S101, one embodiment is as follows: Figure 2 As shown, the step of preprocessing the real-time image sequence based on the image characteristics to obtain the target image sequence includes: S1011. Process each frame of the real-time image in the real-time image sequence according to the processing method corresponding to the image characteristics to obtain a binary image including dynamic water mist region and non-dynamic water mist region; S1012. Filter the dynamic water mist region in the binary image corresponding to each real-time image to obtain the target image sequence composed of images corresponding to non-dynamic water mist regions.

[0028] In steps S1011-S1012, since dynamic water mist is a high-frequency signal in the image, its grayscale changes drastically and its frequency of change is much higher than that of water or a water container, the real-time image sequence needs to be grayscale processed first. Then, each frame of the real-time image sequence is processed according to a preset processing method, specifically the inter-frame difference method, to identify the dynamic water mist in each frame of the real-time image. Since dynamic water mist affects the accuracy of water-adding action recognition, the dynamic water mist region in the binary image corresponding to each frame of the real-time image is filtered by formula (1) to obtain the target image sequence composed of images corresponding to non-dynamic water mist regions. Formula (1) is expressed as: (1); Where I(t) is the current grayscale frame, I(t-1) is the previous grayscale frame, α is the experimentally calibrated water mist weight coefficient, which can take a value of 0.3, and M fog(t) The binary image generated by high-frequency texture analysis is then processed using formula (1) to remove dynamic water mist, thus ensuring the accuracy of the image recognition technology. In a specific implementation of step S1011, one embodiment is as follows: Each frame of the real-time image in the real-time image sequence is processed according to the processing method corresponding to the image characteristics to obtain a binary image including dynamic and non-dynamic water mist regions, including: S10111 Perform frequency domain analysis on each frame of real-time image to distinguish the high-frequency region and low-frequency region in each frame of real-time image; S10112. Based on the features of the dynamic water mist region, extract the high-frequency region in each frame of real-time image to distinguish the dynamic water mist region from the non-dynamic water mist region.

[0029] In step S10111- In S10112, the control module performs frequency domain analysis on each grayscale real-time image based on the FFT Fourier transform algorithm. Then, based on the characteristic that dynamic water mist is a high-frequency signal in the image, with drastic grayscale changes and a frequency much higher than that of water or water containers, it distinguishes the high-frequency and low-frequency regions in the frequency domain. Then, it transforms each real-time image from the spatial domain to the frequency domain F(u, v), uses a high-pass filter to extract the high-frequency region, and filters out the low-frequency to obtain the inverse Fhigh(u, v). Then, it uses the FFT inverse Fourier transform to transform back to the spatial domain to obtain the high-frequency image Ihign. At this point, the high-frequency image Ihign has become a state where the outlines of water mist and non-water mist are clearly distinguished. Then, for the high-frequency image Ihign, a threshold can be found through statistical analysis. Values ​​greater than T are 1, representing dynamic water mist regions, and values ​​less than T are 0, representing non-dynamic water mist regions. In this way, the matrix becomes a state of 0 and 1, clearly indicating where the dynamic water mist is and where the non-dynamic water mist is, thus obtaining the dynamic water mist regions and non-dynamic water mist regions, and thus identifying the dynamic water mist in each real-time image.

[0030] In step S102, the control module controls a pre-trained water-addition recognition model based on the MobileNet-SSD model. The preprocessing module described above is a branch of the water-addition recognition model; dynamic water mist removal is also a function of the model. The water-addition action is identified based on the target bounding box in the target image sequence. This action consists of a water container, a hand, and water flow, and these elements have a specific positional relationship. For example, the hand is held in relation to the water container, and the water flow is connected to the water container's outlet. The target bounding box simultaneously frames the area containing the water container, hand, and water flow in each frame of the target image sequence. The water container, hand, and water flow are identified by the water-addition recognition model, which outputs the coordinates (x, y, y) of the target bounding box. min ,y min ,x max ,y max The confidence level and category are used to determine the probability of water addition and the location information of the target bounding box in the target image sequence. This involves statistically analyzing and summarizing the location information of the target bounding box in each frame of the target image to observe the positional changes of the target bounding box. Based on the location information of the target bounding box in each frame of the target image, the target distance from the target bounding box to the air outlet of the humidifier is calculated. Based on the target distance, it is possible to accurately determine whether the water addition action corresponding to the target bounding box is close to or far from the air outlet of the humidifier. The target distance is calculated based on formula (2). (2); Where kd =0.05cm / pixel, (xc,yc) are the center coordinates of the target bounding box.

[0031] The water addition recognition model was trained using the following method: a) Constructing the dataset i. Data source: CMOS camera of the same specifications as the humidifier (640x480, 15fps); scene settings include different fog output states, different lighting conditions, and different water filling actions (holding a kettle, cup, bottle, or directly pouring).

[0032] ii. Data categories: Hands (2000 images, different skin tones, holding, extending, with water mist in the background); Containers (2000 images, water jugs, cups, bottles of different materials, including partial occlusion); Water flow (1500 images, continuous water droplets, different pouring speeds); Negative samples (1000 images, mobile phones, books, and other unrelated objects, hand gestures without the intention to add water).

[0033] iii. Labeling method: Manually label bounding boxes using the LabelImg tool in VOC (XML) format, labeling the category and coordinates. Label 1-3 targets per image to ensure diversity.

[0034] iv. Data augmentation: random rotation (±15°), scaling (0.8-1.2x), brightness adjustment (±20%), addition of Gaussian noise (σ=0.05), expanding to a total of 10,000 images.

[0035] b) Model selection and optimization: i. The water addition recognition model uses the lightweight MobileNet-SSD suitable for embedded devices; the last convolutional layer is replaced with a depthwise separable convolution to reduce computation; a water mist feature branch is added: a wavelet transform module is connected in parallel after the feature extraction layer to extract high-frequency water mist texture and enhance anti-interference ability.

[0036] ii. Output layer: Number of categories: 4 (hand, water container, water flow, background); bounding box, confidence score, and IoU.

[0037] c) Training process: i. Training platform: PC (GPU: NVIDIA RTX 3060, 12GB VRAM), export to MCU after completion.

[0038] ii. Parameter settings: Learning rate: Initially 0.001, reduced to 0.0001 in round 50, using a cosine annealing strategy.

[0039] Batch size: 16 (to balance video memory and convergence speed).

[0040] Optimizer: Adam (β1=0.9, β2=0.999), accelerates gradient descent.

[0041] Loss function: Classification loss: cross-entropy loss (Softmax); localization loss: Smooth L1; total loss: L = Lcls + λLloc (λ = 1.0).

[0042] Number of iterations: 15,000 (approximately 50 epochs), with a checkpoint saved every 500 iterations.

[0043] iii. Data partitioning: Training set: 70% (7000 images); Validation set: 20% (2000 images); Test set: 10% (1000 images).

[0044] iv. Training steps: Pre-training: The MobileNet backbone is initialized using ImageNet pre-trained weights.

[0045] Fine-tuning: Freeze the first 10 layers, train only the later layers and the water mist feature branch, 5 epochs; unfreeze all layers and continue training until convergence.

[0046] Pruning optimization: Remove channels with confidence < 0.1, reducing the number of parameters from 1.2M to 0.9M.

[0047] d) Verification and Adjustment i. Evaluation indicators: Mean accuracy (mAP): IoU=0.5, target mAP>0.85.

[0048] Detection speed: Single frame inference <50ms (STM32 test).

[0049] False positive rate: <5%, avoiding triggering of irrelevant actions.

[0050] ii. Verification results (example): Hand mAP: 0.92; Water container mAP: 0.89; Water flow mAP: 0.87.

[0051] Average inference time: 45ms.

[0052] iii. Adjustment strategy: If the water flow mAP < 0.85, increase the water flow samples (+500 images) and adjust the loss weight (λ=1.2); if the false positive rate > 5%, increase the confidence threshold (from 0.85 to 0.9).

[0053] e) Embedded deployment i. Model conversion: Convert to TensorFlow Lite format.

[0054] ii. Deployment process: Burn it into the STM32 Flash and run it in conjunction with the CMSIS-NN library.

[0055] In step S103, after the control module obtains the position information of the target bounding box of each frame of the target image, it can obtain multiple position information of the target bounding box in the target image sequence. Based on the principle that seven points can be found in the Cartesian coordinate system to fit a curve of a quadratic motion trajectory, in this application, the position information of the target bounding box corresponding to 7 or more frames of the target image in the target image sequence, that is, the center coordinate points of the target bounding box, are fitted by the least squares method to obtain the water addition motion trajectory. The specifically obtained water addition motion trajectory is expressed by formula (3) as follows: (3); Where a, b, and c are all obtained by fitting with the least squares method, and t is time. Based on the water addition motion trajectory, the future position relationship between the target bounding box and the air outlet in the future time period can be predicted, so as to predict whether the target bounding box is approaching the air outlet of the humidifier, ensuring the accuracy of the prediction.

[0056] In the specific implementation process of step S103, there is an embodiment: The predicting the future position relationship between the target bounding box and the air outlet in the future time period based on the water addition motion trajectory includes: S1031. Predict the future height data of the target bounding box in a preset future time period through the water addition motion trajectory; S1032. Compare the future height data with the previously obtained height data of the air outlet to determine the future position relationship.

[0057] In steps S1031 - S1032, through the water addition motion trajectory, when any time period is input, the corresponding height data of any time period can be obtained. If the input time is a future time period, the future height data of the target bounding box in the preset future time period can be predicted. For example, if the preset future time period is 0.5 s after the current moment, then t in formula (3) is replaced with t + 0.5, and the obtained y (t+0.5) is the future height data. Compare the future height data with the previously obtained height data of the air outlet. If y (t+0.5) < y0, the future position relationship between the target bounding box and the air outlet of the humidifier is approaching, otherwise y (t+0.5) ≥ y0, the future position relationship between the target bounding box and the air outlet of the humidifier is away.

[0058] S104. The control module fuses the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water-adding intention. Specifically, it determines the user's intention to add water by analyzing the hand, water container, water flow, future positional relationship, and target distance in multiple consecutive frames of the target image sequence. This intention indicates a high probability that the water outlet of the humidifier will be filled with water in the next moment. Since the humidifier's air outlet is currently humidifying, the resulting water mist could seriously affect the user's safety. To ensure the user's safety during water addition... Based on the water-adding intention, a stop command is generated to control the water-adding actuator to execute the stop command and stop humidification, i.e., turn off the fan or atomizer, thereby ensuring the user's safety when adding water. In order to avoid the occurrence of momentary misjudgment that affects the accuracy of water-adding intention judgment, after confirming that the water-adding intention is obtained based on the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet, the control module outputs a signal "1" and continues to confirm for 0.5 seconds, thereby ensuring that the user can easily add water through the air outlet of the humidifier, thus ensuring the user's safety and normal use of the equipment.

[0059] After the humidifier stops, the water filling mechanism performs a stop detection every second. If D>20 cm and there is no water flow within 15 seconds after the stop detection, the water filling mechanism resumes operation.

[0060] In the specific implementation of step S104, one embodiment is as follows: Figure 3 As shown, the control module integrates the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water filling intention, including: S1041. Statistically analyze the evaluation data corresponding to different combinations of water-pouring container, hand, and water flow in the target image within a consecutive frame number; wherein the water-pouring container, hand, and water flow also constitute a water-adding action; S1042. Based on the evaluation data corresponding to the different combinations, the numerical comparison results of the target distance and the corresponding adaptive threshold, and the future positional relationship, the intention to add water is determined.

[0061] In steps S1041-S1042, the control module statistically analyzes the evaluation data corresponding to different combinations of the water container, hand, and water flow in the target image within a consecutive number of frames. The consecutive number of frames can be set according to actual conditions; in this application, it is set to 5 frames, approximately 0.33 seconds. That is, the control module considers the water container and hand as one combination and the water flow as another combination, and statistically analyzes the corresponding evaluation data for these two combinations. The evaluation data is represented by a confidence level (mAP). If the confidence level (mAP) corresponding to the water container and hand or the water flow within a consecutive number of frames is greater than the corresponding confidence threshold, and the target distance... If the comparison result between the target distance and the corresponding adaptive threshold is less than the corresponding adaptive threshold, and the future position relationship is close, then it is determined that the user has generated the intention to add water. The water container, hand, and water flow also constitute the water adding action. The accuracy of the obtained water adding intention is further reflected based on the water adding action. The confidence thresholds corresponding to the confidence mAP of the water container, hand, and water flow are set according to actual settings. In this application, the confidence thresholds are set to 0.85 and 0.7, respectively. At this time, the water adding shutdown mechanism is triggered based on the water adding intention, and a shutdown command is generated based on the water adding intention.

[0062] In practical use, if a user holds a water bottle close to the object, and the current distance (D) is 9 cm, the distance (D) is calculated after four consecutive frames of detection. th =9.6 cm, N th =5, then the result is D. th <9.6 but insufficient frames, so it will not be triggered temporarily, pending confirmation in the next frame; the control module also has an enhancement mechanism, if the confidence level mAP of the water flow is >0.7 and the trajectory is downward, meaning the future positional relationship is close, even if D>D th The water supply shutdown mechanism was still triggered.

[0063] In a specific implementation of step S1042, one embodiment is as follows: the adaptive threshold corresponding to the target distance is determined based on the following steps, including: A1. Obtain the historical target images and corresponding historical target distances corresponding to the multiple historical water filling and shutdown mechanisms of the humidifier, and calculate them respectively to obtain the calculation results; A2. Based on the calculation results, adjust the initial threshold of the target distance to obtain the adaptive threshold.

[0064] In steps A1-A2, after triggering the water filling and shutdown mechanism, the control module acquires historical target images and corresponding historical target distances corresponding to multiple historical water filling and shutdown mechanisms of the humidifier, and calculates them respectively to obtain the calculation results. This application sets up the acquisition of historical target images and historical target distances corresponding to the water filling events of the most recent 10 water filling and shutdown mechanisms. N represents the number of frames in the historical target image where the target bounding box was detected, and D represents the historical target distance. hist

[10] represents the 10 sets of water-addition events stored, N hist

[10] Stored in the flash or RAM of the control module, then calculated based on formulas (4)-(5): (4); (5); Among them, median(D) hist ): The median of historical target distances, reflecting typical water filling distances; σ(D) hist ): Standard deviation of historical target distance, measuring fluctuations in user behavior; mean(N) hist ): Average of historical frame counts, reflecting the continuity of user actions; round: rounds down to accommodate discrete frame counts; 0.5: safety factor, ensuring the threshold is slightly higher than the median to avoid missed detections, and also to set the threshold at a moderately high level to avoid excessive deviation and missed counts; initial threshold: D th =10cm, N th =5, and its update frequency is once every 5 water additions, and it is stored in Flash. For example, the initial threshold: D th =10, N th =5; Water additions 1-5: Data: D hist =[8,9,7,10,8],N hist =[4,5,4,6,5]; Calculate: median(D) hist )=8; σ(Dhist)≈1.0 (simplified calculation); D th =8 + 0.5 • 1 = 8.5; mean(N hist ) = (4 + 5 + 4 + 6 + 5) / 5 = 4.8, N th =round(4.8)=5; Update: D th =8.5 cm, N th =5; Water additions for the 6th-10th times: D hist=[8,9,7,10,8,9,11,7,10,9]; Calculate: median(D) hist )=9,σ(D hist )≈1.2; D th =9 + 0.5 • 1.2 = 9.6 cm, then update D th N th The same applies to the update.

[0065] The water filling control system gradually adapts to the user's individual habits, and the adaptive threshold is adjusted from 10cm to 9.6cm, which is more in line with reality.

[0066] In a specific implementation of step S1042, one embodiment is as follows: after determining the water-adding intention based on the evaluation data, the target distance and the corresponding adaptive threshold, and the future positional relationship, the process includes: B1. Determine whether the evaluation data of the hand and the target distance meet the preset abnormal water filling conditions and generate the corresponding judgment result; B2. Based on the judgment result, trigger the water filling and shutdown mechanism to control the humidifier to stop responding to the water filling and shutdown mechanism.

[0067] In steps B1-B2, it is determined whether the evaluation data of the hand and the target distance meet the preset abnormal water filling conditions and a corresponding judgment result is generated. That is, if the confidence level of the hand is >0.9 and D<5cm, the judgment result is that a child is adding water to the humidifier. Since the air outlet of the humidifier is in a humidifying state at this time, the generated water mist seriously affects the safety of the child. In order to ensure the safety of the child when adding water, the humidifier triggers the water filling shutdown mechanism to control the humidifier to stop responding to the water filling shutdown mechanism and issue an audible and visual alarm to remind adults to ensure the safety of the child.

[0068] Example 2 This application also provides a water addition detection device based on image recognition, such as... Figure 4 The diagram shows a block diagram of a water filling detection device based on image recognition. This device performs functions corresponding to the steps of executing a water filling detection method based on image recognition on a terminal device as described above. The device can be understood as a server component including a processor. The water filling detection device described in this application is suitable for a water filling control system, which includes an image acquisition module, a control module, and a water filling actuator. The device includes: The acquisition module 401 is used to acquire real-time image sequences above the air outlet of the humidifier in real time, obtain the image characteristics of the real-time image sequence through the control module, and preprocess the real-time image sequence based on the image characteristics to obtain the target image sequence. The control module 402 is used to control the pre-trained water addition recognition model to recognize the water addition action based on the target bounding box in the target image sequence, and to obtain multiple positional information of the target bounding box in the target image sequence. The fitting module 403 is used by the control module to fit the water-adding motion trajectory based on the multiple position information, and to predict the future positional relationship between the target bounding box and the air outlet in the future time period based on the water-adding motion trajectory. The water filling module 404 is used by the control module to fuse the target image sequence, the future positional relationship, and the target distance between the target bounding box and the air outlet to obtain the water filling intention, generate a stop command based on the water filling intention, and control the water filling actuator to execute the stop command to perform safe water filling.

[0069] In one feasible implementation, the water supply module includes: The statistics module is used to collect evaluation data corresponding to different combinations of water-pouring container, hand, and water flow in the target image within a consecutive number of frames; wherein the water-pouring container, hand, and water flow also constitute a water-adding action; The determination module is used to determine the intention to add water based on the evaluation data corresponding to the different combinations, the numerical comparison results of the target distance and the corresponding adaptive threshold, and the future positional relationship.

[0070] In one feasible implementation, the water supply module further includes: The acquisition module is used to acquire historical target images and corresponding historical target distances corresponding to multiple historical water filling and shutdown mechanisms of the humidifier, and perform calculations to obtain the calculation results. An adjustment module is used to adjust the initial threshold of the target distance based on the calculation results to obtain the adaptive threshold.

[0071] In one feasible implementation, the water supply module also includes: The judgment module is used to judge whether the evaluation data of the hand and the target distance meet the preset abnormal water filling conditions and generate the corresponding judgment result; The triggering module is used to trigger the water filling and shutdown mechanism based on the judgment result, so as to control the humidifier to stop responding to the water filling and shutdown mechanism.

[0072] In one feasible implementation, the acquisition module includes: The processing module is used to process each frame of the real-time image in the real-time image sequence according to the processing method corresponding to the image characteristics, so as to obtain a binary image including dynamic water mist region and non-dynamic water mist region; The filtering module is used to filter the dynamic water mist region in the binary image corresponding to each frame of real-time image, so as to obtain the target image sequence composed of the images corresponding to the non-dynamic water mist region.

[0073] In one feasible implementation, the acquisition module further includes: The analysis module is used to perform frequency domain analysis on each frame of the real-time image to distinguish the high-frequency region and the low-frequency region in each frame of the real-time image. The extraction module is used to extract high-frequency regions in each frame of real-time image based on the features of the dynamic water mist region, so as to distinguish between the dynamic water mist region and the non-dynamic water mist region.

[0074] In one feasible implementation, the fitting module includes: The prediction module is used to predict the future height data of the target bounding box within a preset future time period based on the water-adding motion trajectory. The comparison module is used to compare the future height data with the pre-acquired height data of the air outlet to determine the future positional relationship.

[0075] Example 3 This application also provides an electronic device, such as Figure 5 As shown, it includes: a processor 501, a memory 502, and a bus 503. The memory 502 stores machine-readable instructions that can be executed by the processor 501. When the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503. When the machine-readable instructions are executed by the processor 501, the steps of any of the image recognition-based water addition detection methods described above are executed.

[0076] Example 4 This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the image recognition-based water addition detection methods described above.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0078] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0081] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image recognition-based water addition detection method, characterized by, The method is suitable for a water adding control system including an image acquisition module, a control module and a water adding execution mechanism, and the method includes: The image acquisition module acquires a real-time image sequence above an air outlet of a humidifier in real time, obtains image features of the real-time image sequence through the control module, and pre-processes the real-time image sequence based on the image features to obtain a target image sequence; The control module controls a pre-trained water adding recognition model to recognize a water adding action based on a target bounding box in the target image sequence, and obtains a plurality of position information of the target bounding box in the target image sequence; The control module fits a water adding motion trajectory based on the plurality of position information, and predicts a future position relationship between the target bounding box and the air outlet in a future time period based on the water adding motion trajectory; The control module fuses the target image sequence, the future position relationship and a target distance between the target bounding box and the air outlet to obtain a water adding intention, generates a shutdown instruction based on the water adding intention, and controls the water adding execution mechanism to execute the shutdown instruction to perform safe water adding.

2. The method of claim 1, wherein, The control module fuses the target image sequence, the future position relationship and a target distance between the target bounding box and the air outlet to obtain a water adding intention, including: Statistics of different combinations of the target image in the continuous frame number, the water container, the hand and the water flow respectively correspond to the evaluation data; wherein the water container, the hand and the water flow also constitute a water adding action; Based on the evaluation data corresponding to the different combinations, the numerical comparison result of the target distance and the corresponding adaptive threshold value, and the future position relationship, the water adding intention is determined.

3. The method of claim 2, wherein, The adaptive threshold value corresponding to the target distance is determined based on the following steps: Obtain the historical target image corresponding to the multiple historical water adding shutdown mechanisms of the humidifier and the corresponding historical target distance, and calculate them respectively to obtain the calculation result; Adjust the initial threshold value of the target distance based on the calculation result to obtain the adaptive threshold value.

4. The method of claim 2, wherein, After determining the water adding intention based on the evaluation data, the target distance and the corresponding adaptive threshold value, and the future position relationship, including: Determine whether the evaluation data of the hand and the target distance meet the preset water adding abnormal condition and generate a corresponding determination result; Trigger the water adding shutdown mechanism based on the determination result to control the humidifier to shut down in response to the water adding shutdown mechanism.

5. The method of claim 1, wherein, The method includes: According to the processing mode corresponding to the image features, each real-time image in the real-time image sequence is processed to obtain a binary image including a dynamic water mist area and a non-dynamic water mist area; Filter the dynamic water mist area in the binary image corresponding to each real-time image to obtain a target image sequence composed of images corresponding to the non-dynamic water mist area.

6. The method of claim 5, wherein, According to the processing mode corresponding to the image features, each real-time image in the real-time image sequence is processed to obtain a binary image including a dynamic water mist area and a non-dynamic water mist area, including: Performing frequency domain analysis on each of the real-time images to distinguish high-frequency regions and low-frequency regions in each of the real-time images; Extracting the high-frequency regions in each of the real-time images based on the features of the dynamic water mist region to distinguish the dynamic water mist region and the non-dynamic water mist region.

7. The method of claim 1, wherein, The future position relationship between the target bounding box and the air outlet in the future time period is predicted based on the water adding motion trajectory, which includes: The future height data of the target bounding box in the preset future time period is predicted through the water adding motion trajectory; The future height data is compared with the height data of the air outlet obtained in advance to determine the future position relationship.

8. An image recognition-based water addition detection device, characterized by, The device is suitable for a water adding control system, which includes an image acquisition module, a control module and a water adding execution mechanism, and the device includes: The acquisition module is used for the image acquisition module to collect a real-time image sequence above an air outlet of a humidifier in real time, and the control module is used to obtain image features of the real-time image sequence and pre-process the real-time image sequence based on the image features to obtain a target image sequence; The control module is used to control a pre-trained water adding recognition model to recognize a water adding action based on a target bounding box in the target image sequence and obtain a plurality of position information of the target bounding box in the target image sequence; The fitting module is used to fit a water adding motion trajectory based on the plurality of position information by the control module, and predict a future position relationship between the target bounding box and the air outlet in a future time period based on the water adding motion trajectory; The water adding module is used to fuse a water adding intention obtained by the control module from the target image sequence, the future position relationship and a target distance between the target bounding box and the air outlet, generate a shutdown instruction based on the water adding intention, and control the water adding execution mechanism to execute the shutdown instruction to perform safe water adding.

9. An electronic device, comprising: The device includes: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the water adding detection method based on image recognition in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the water adding detection method based on image recognition in any one of claims 1 to 7.

Images