A noise processing method and device, intelligent device and storage medium

Abstract

Embodiments of the present application disclose a noise processing method and device, an intelligent device and a storage medium. Embodiments of the present application can acquire a video of running of at least one object in a device; perform centroid tracking on the at least one object according to the video to generate a centroid tracking trajectory of each object; determine a target object in which noise occurs in the device, and a centroid position of the target object and a target time point at which the target object occurs noise according to the centroid tracking trajectory; acquire target noise emitted by the target object and noise information of the target noise at the target time point; and perform noise directional processing on the target noise according to the noise information, the centroid position and the target time point. The embodiments of the present application solve the problem of inaccurate noise processing of an intelligent device and improve the accuracy of noise processing of the intelligent device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a noise processing method and device, intelligent equipment and storage medium. BACKGROUND

[0002] Some existing devices may have high noise due to large movement offset of internal components during operation. For example, the washing machine device has motor running noise, and the inner drum rotates at high speed, so that the noise is too large in the washing and dehydration state, thereby affecting the user experience. The existing device noise reduction scheme is mostly passive noise reduction through physics. For example, noise reduction is achieved by applying isolation plates, sound insulation cotton and reinforcing materials, etc. However, the above-mentioned method needs to add sound insulation materials, the noise reduction is not accurate, and the noise reduction effect is not ideal. SUMMARY

[0003] The embodiments of the present application provide a noise processing method, device, intelligent equipment and storage medium, which can improve the accuracy of noise processing of intelligent equipment.

[0004] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0005] The embodiments of the present application provide a noise processing method, which comprises:

[0006] Obtaining a video of at least one object running in a device;

[0007] Tracking the center of mass of the at least one object according to the video to generate a center of mass tracking trajectory of each object;

[0008] Determining a target object that generates noise in the device, a center of mass position of the target object and a target time point at which the target object generates noise according to the center of mass tracking trajectory;

[0009] Obtaining a target noise generated by the target object and noise information of the target noise at the target time point;

[0010] According to the noise information, the center of mass position and the target time point, performing noise directional processing on the target noise.

[0011] In an embodiment, the tracking of the center of mass of the at least one object according to the video to generate the center of mass tracking trajectory of each object comprises:

[0012] Frame processing the video to obtain a plurality of picture sequences;

[0013] Arranging the plurality of picture sequences in time sequence into a plurality of continuous frame pictures;

[0014] perform object detection on each of the plurality of continuous frame pictures to determine a centroid position of the at least one object in each of the plurality of continuous frame pictures;

[0015] track a trajectory of the at least one object according to the centroid positions in an order of the plurality of continuous frame pictures to generate a centroid tracking trajectory of the at least one object in the plurality of continuous frame pictures.

[0016] In an embodiment, the object includes a matched object, and the tracking a trajectory of the at least one object according to the centroid positions in an order of the plurality of continuous frame pictures to generate a centroid tracking trajectory of the at least one object in the plurality of continuous frame pictures includes:

[0017] obtaining the centroid position of the at least one object in each of the plurality of continuous frame pictures;

[0018] matching each object according to the centroid position of each object in adjacent frame pictures to determine a matched object, to obtain the matched object;

[0019] generating a centroid tracking trajectory of the matched object in a time sequence according to the centroid position of the matched object, the centroid tracking trajectory of the matched object being a tracking trajectory of the centroid of the matched object in the plurality of continuous frame pictures.

[0020] In an embodiment, the matching each object according to the centroid position of each object in adjacent frame pictures to determine a matched object, to obtain the matched object includes:

[0021] obtaining a centroid distance of each object between each adjacent frame picture;

[0022] determining whether the centroid distance satisfies a first preset condition;

[0023] if the centroid distance satisfies the first preset condition, determining that the object between the adjacent frame pictures is a matched object.

[0024] In an embodiment, the determining the target object in which noise occurs in the device according to the centroid tracking trajectory, and a centroid position of the target object and a target time point in which noise occurs includes:

[0025] obtaining a centroid distance of the matched object between each adjacent frame picture from the centroid tracking trajectory of the matched object;

[0026] if the centroid distance of the matched object between adjacent frame pictures satisfies a second preset condition, determining that the matched object is a target object in which noise occurs, and obtaining a centroid position of the target object;

[0027] A time point corresponding to a latter one of the adjacent frame pictures is determined as a target time point at which the noise occurs.

[0028] In an embodiment, the objects include new objects, and the center point tracking of the at least one object according to the video to generate the center point tracking trajectory of each object includes:

[0029] The number of objects in the adjacent frame pictures is obtained.

[0030] If the number of objects in the latter one of the adjacent frame pictures is greater than the number of objects in the former one of the adjacent frame pictures, and the center point distance of the objects between the adjacent frame pictures does not satisfy a first preset condition, it is determined that a new object exists in the latter one of the adjacent frame pictures.

[0031] The center point position of the new object is obtained.

[0032] The center point tracking trajectory of the new object is generated with the center point position of the new object as an initial position.

[0033] In an embodiment, the objects include disappeared objects, and after the center point tracking of the at least one object according to the video to generate the center point tracking trajectory of each object, the method includes:

[0034] If the object in the frame picture does not appear a matched object in a subsequent frame picture, it is determined that the object is a disappeared object.

[0035] The center point tracking trajectory of the disappeared object is deleted.

[0036] In an embodiment, the object detection of each frame picture in the plurality of continuous frame pictures to determine the center point position of the at least one object in each frame picture includes:

[0037] The contour information of the at least one object in each frame picture is detected.

[0038] The bounding box of the at least one object is determined according to the contour information.

[0039] The center point position of the at least one object is determined according to the bounding box.

[0040] In an embodiment, the noise directional processing of the target noise according to the noise information, the center point position, and the target time point includes:

[0041] The noise direction of the target noise is determined according to the center point position.

[0042] The noise processing signal is generated according to the noise direction and the target noise.

[0043] According to the noise processing signal, the target noise is processed at the target time point.

[0044] In an embodiment, the generating of the noise processing signal according to the noise direction and the target noise signal comprises:

[0045] Obtaining noise information of the target noise;

[0046] According to the noise information, determining a signal parameter of the noise processing signal;

[0047] According to the signal parameter, generating the noise processing signal.

[0048] According to an aspect of the present application, a noise processing device is also provided, comprising:

[0049] A first obtaining module is configured to obtain a video of at least one object running inside a device;

[0050] A generating module is configured to perform centroid tracking on the at least one object according to the video, to generate a centroid tracking trajectory of each object;

[0051] A determining module is configured to determine a target object in which noise occurs in the device, and a centroid position of the target object and a target time point at which the noise occurs, according to the centroid tracking trajectory;

[0052] A second obtaining module is configured to obtain a target noise emitted by the target object at the target time point, and noise information of the target noise;

[0053] A processing module is configured to perform noise directional processing on the target noise according to the noise information, the centroid position and the target time point.

[0054] According to an aspect of the present application, an intelligent device is also provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor invokes the computer program in the memory to execute any noise processing method provided in the embodiments of the present application.

[0055] According to an aspect of the present application, a storage medium is also provided, which is configured to store a computer program, and the computer program is loaded by a processor to execute any noise processing method provided in the embodiments of the present application.

[0056] The embodiment of the present application can acquire a video of running of at least one object in a device; perform centroid tracking on the at least one object according to the video to generate a centroid tracking trajectory of each object; determine a target object in which noise occurs in the device, and a centroid position of the target object and a target time point at which the target object occurs noise according to the centroid tracking trajectory; acquire target noise emitted by the target object and noise information of the target noise at the target time point; and perform noise directional processing on the target noise according to the noise information, the centroid position and the target time point. In this way, the intelligent device can determine the centroid position of the target object in which noise occurs, the target noise and the target time point at which the target noise occurs according to the centroid tracking trajectory of the object when the object in the device emits noise, so as to perform noise directional processing on the target noise at the target time point, thereby accurately performing active noise reduction processing on the noise and improving the accuracy of noise processing. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0058] Figure 1 is a flowchart of a noise processing method provided by the embodiment of the present application;

[0059] Figure 2 is another flowchart of a noise processing method provided by the embodiment of the present application;

[0060] Figure 3 is another flowchart of a noise processing method provided by the embodiment of the present application;

[0061] Figure 4 is a schematic diagram of object detection in a picture of a noise processing method provided by the embodiment of the present application;

[0062] Figure 5 is a schematic diagram of centroid distance of objects between adjacent frame pictures in a noise processing method provided by the embodiment of the present application;

[0063] Figure 6 is a schematic diagram of processing target noise in a noise processing method provided by the embodiment of the present application;

[0064] Figure 7 is a schematic diagram of a noise processing device provided by the embodiment of the present application;

[0065] Figure 8 is a structural schematic diagram of an intelligent device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0067] The embodiments of the present application provide a noise processing method and device, an intelligent device and a storage medium (i.e., a computer readable storage medium). The noise processing method can be applied to a noise processing device, which can be integrated in an intelligent device. The intelligent device can be connected to a server or a terminal in communication, and the server can be a physical server, a server cluster composed of multiple physical servers or a distributed system, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, but is not limited thereto. The server and the terminal can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. The terminal can be a mobile phone, a computer, a household appliance, or a wearable device.

[0068] The following will be described in detail. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0069] In the present embodiment, the description will be made from the perspective of the intelligent device. Please refer to Figure 1 , Figure 1 FIG. 1 is a flow diagram of a noise processing method according to an embodiment of the present application. The noise processing method can include the following steps.

[0070] S101, acquiring a video of at least one object running in a device.

[0071] The device can be an intelligent device that generates noise when internal components run, such as a smart washing machine, a smart fan, a smart air conditioner, a smart vehicle, etc. The object can be an internal component that may generate noise when the device runs, such as a motor, a clutch, and a dehydration inner tube in a smart washing machine, or a motor or a motor in a smart fan, a smart air conditioner, or a smart vehicle. The object can be one or more. The video can record the continuous motion trajectory of at least one object during the running process in real time.

[0072] In the embodiment, the device can be provided with a camera for collecting a video of at least one object running inside the device. The camera can be a time of flight (TOF) camera, which can calculate the transmission time of light from a light source to a photographed object to calculate the depth information of the photographed object. The execution subject of the noise processing method in the embodiment can be a smart device.

[0073] For example, the smart device collects a video of at least one object running inside the device through the camera provided inside the device and stores the video in the local or server. The smart device can obtain the video of at least one object running inside the device from the local or server, or the smart device directly obtains the video after collecting the video of at least one object running inside the device through the camera.

[0074] S102, center tracking of at least one object is performed according to the video to generate a center tracking trajectory of each object.

[0075] The center can be the center of the bounding box of the at least one object detected in the video. The center tracking trajectory can be the motion trajectory of the center of each object in the continuous video, that is, the moving trajectory of the position of the center in the multiple continuous frame pictures with the change of time of the video.

[0076] For example, the smart device detects the bounding box of at least one object in the video, calculates the center position of each object according to the bounding box of each object, and generates the center tracking trajectory of each object according to the motion trajectory of the center position in the video, so that the smart device can judge whether the object in the video appears noise based on the center tracking trajectory.

[0077] S103, a target object appearing noise in the device is determined according to the center tracking trajectory, and the center position of the target object and the target time point appearing noise.

[0078] The target object appearing noise can be an object whose center position produces a large distance movement in the center tracking trajectory. The center position can be the center coordinates or the vector of the center relative to the noise processing device. The target time point appearing noise can be the time point of the video frame corresponding to the time when the center position of the target object produces a large distance movement.

[0079] For example, the change of the center position of the object in the video is judged according to the center tracking trajectory. If the change of the center position suddenly increases, the object is determined as the target object appearing noise, and the center position when the center position of the target object produces a large distance movement and the corresponding target time point are obtained.

[0080] S104, acquire target noise emitted by the target object and noise information of the target noise at the target time point.

[0081] The noise information of the target noise can include at least one of the number of elements of the noise, the element spacing, the sound speed, and the noise signal frequency.

[0082] For example, the audio receiving and transmitting device, such as a microphone, can be provided in the smart device. The audio receiving and transmitting device can be located at the same position as the camera in the smart device. The smart device can receive the target noise of the target object through the microphone at the target time point when the target object emits the target noise, and acquire at least one of the noise information of the target noise, so as to process the target noise.

[0083] S105, according to the noise information, the centroid position, and the target time point, performing noise directional processing on the target noise.

[0084] The noise directional processing can be directional noise reduction processing on the target noise.

[0085] For example, when the target object emits the target noise, a noise processing signal is generated according to the noise information of the target noise, and the noise processing signal is directionally sent to the centroid position of the target noise, so as to perform noise reduction processing on the target noise.

[0086] In the technical scheme provided in this embodiment, a video of at least one object running in the device is acquired; centroid tracking is performed on the at least one object according to the video to generate a centroid tracking trajectory of each object; a target object that emits noise in the device is determined according to the centroid tracking trajectory, as well as a centroid position of the target object and a target time point at which the target object emits the noise; target noise emitted by the target object and noise information of the target noise are acquired at the target time point; and noise directional processing is performed on the target noise according to the noise information, the centroid position, and the target time point. In this way, when an object in the device emits noise, the smart device can determine the centroid position of the target object that emits the noise, the target noise, and the target time point at which the target object emits the noise according to the centroid tracking trajectory of the object, so as to perform noise directional processing on the target noise at the target time point, thereby accurately performing active noise reduction processing on the noise and improving the accuracy of noise processing.

[0087] Based on the above embodiment, please refer to Figure 2 , Figure 2 Another flowchart of the noise processing method provided in this embodiment is provided. The centroid tracking is performed on the at least one object according to the video to generate the centroid tracking trajectory of each object, which can include:

[0088] S201, frame processing is performed on the video to obtain a plurality of picture sequences.

[0089] The video can record the continuous motion trajectory of at least one object in the running process in real time. Each frame of picture can contain at least one object.

[0090] For example, the frame rate of the video is obtained, and the video is frame-processed according to the frame rate of the video to obtain a plurality of picture sequences. In an actual application scenario, the smart device can collect the video of the running of at least one object through the camera device, and divide the video into a plurality of picture sequences according to the video frame rate, so as to track the center of mass of the object in the plurality of picture sequences.

[0091] S202, arrange the plurality of pictures in time sequence as a plurality of continuous frame pictures.

[0092] For example, picture A, picture B and picture C are arranged in time sequence as first frame picture A, second frame picture B and third frame picture C.

[0093] S203, perform object detection on each frame of picture in the plurality of continuous frame pictures to determine the center of mass position of at least one object in each frame of picture.

[0094] The object can be an internal device that may generate noise when the device is running, such as a motor, a clutch and a dehydration inner cylinder in a smart washing machine, or a motor or a device in a smart fan, a smart air conditioner or a smart vehicle, and the object can be one or more. The center of mass position can be the coordinates of the center of mass relative to the camera device or the vector of the center of mass relative to the camera device.

[0095] For example, refer to Figure 4 , Figure 4 The picture in the noise processing method is shown in the object detection diagram. First, the bounding box of the object in the first frame picture is detected, and the center of mass position of the object is calculated according to the bounding box of the object, and unique identification information is created for each object in the first frame picture.

[0096] Optionally, the object detection on each frame of picture in the plurality of continuous frame pictures to determine the center of mass position of at least one object in each frame of picture can include:

[0097] S1, detecting the contour information of at least one object in each frame of picture;

[0098] S2, determining the bounding box of at least one object according to the contour information;

[0099] S3, determining the center of mass position of at least one object according to the bounding box.

[0100] The contour information of the object can display a boundary shape of the at least one object, which can be a rectangle, or a polygon, or other shapes. The boundary box can be a minimum rectangular box surrounding the contour of the object. The center of mass position can be represented by (x, y) coordinates.

[0101] For example, the intelligent device can detect the video by a preset object detector to generate a boundary box of the object. The preset object detector can be color thresholding + contour extraction, Haar cascade, HOG + linear SVM, SSD, FasterR-CNN, etc. The intelligent device can obtain the coordinates of the rectangular boundary box of the at least one object, calculate the position of the intersection of the diagonal lines of the rectangular boundary box according to the coordinates of the boundary box, and determine the center of mass position of the object.

[0102] S204, according to the center of mass position, track the trajectory of the at least one object in the order of the plurality of continuous frame pictures, and generate a tracking trajectory of the center of mass of the at least one object in the plurality of continuous frame pictures.

[0103] For example, referring to Figure 4 , Figure 4 is a schematic diagram of object detection in a noise processing method. First, the boundary box of the object in the first frame picture is detected, and the center of mass position of the object is calculated according to the boundary box of the object. A unique identification information is created for each object in the first frame picture, and the center of mass position of each object in the first frame picture is taken as the initial position. The center of mass trajectory tracking is performed on each object with unique identification information in the first frame picture in the subsequent frame pictures.

[0104] In an embodiment, according to the center of mass position, the trajectory of the at least one object is tracked in the order of the plurality of continuous frame pictures, and a tracking trajectory of the center of mass of the at least one object in the plurality of continuous frame pictures is generated, which can include:

[0105] S1, obtaining the center of mass position of the at least one object in each frame picture;

[0106] S2, matching each object according to the center of mass position of each object in adjacent frame pictures to determine the matched objects, and obtaining the matched objects.

[0107] The matched objects can refer to the objects with the same identification information in different frame pictures, for example, object A and object B both appear in the first frame picture and the second frame picture, the first frame picture contains object A and object B, and the second frame picture contains object A1 and object B1. The object A in the first frame picture and the object A1 in the second frame picture are matched objects.

[0108] It is understandable that the centroid position of the matched object A1 can be the same as or different from the centroid position of the matched object A. If the centroid position of the matched object A1 is the same as the centroid position of the matched object A, it means that the matched object A has not moved during the operation. If the centroid position of the matched object A1 is different from the centroid position of the matched object A, it means that the matched object A has moved during the operation.

[0109] Optionally, matching each object based on the centroid position of each object in adjacent frame images to determine the matched object may include:

[0110] (1) Obtain the centroid distance of each object between adjacent frames.

[0111] It should be noted that the premise for performing centroid trajectory tracking on at least one object in a video is that the centroid distance between objects in adjacent frames of the video is less than the centroid distance between different objects in the same frame.

[0112] The centroid distance of each object can be the Euclidean distance between the centroid coordinates of each object in adjacent frame images.

[0113] For example, obtain the centroid coordinates of each object in adjacent frames, and calculate the centroid distance between each object in adjacent frames based on the centroid coordinates. (See reference...) Figure 5 , Figure 5 This diagram illustrates the centroid distance between objects in adjacent frames. Objects A1 and B1 are objects in the previous frame of the adjacent frame, while objects A2, B2, and C2 are objects in the next frame of the adjacent frame. The Euclidean distance between objects A1 and A2 can be represented as distance 1, the Euclidean distance between objects A1 and B2 can be represented as distance 2, the Euclidean distance between objects A1 and C2 can be represented as distance 3, the Euclidean distance between objects B1 and B2 can be represented as distance 4, the Euclidean distance between objects B1 and A2 can be represented as distance 5, and the distance between objects B2 and C2 can be represented as distance 6.

[0114] (2) Determine whether the centroid distance meets the first preset condition.

[0115] The first preset condition can be that the centroid distance between an object in the previous frame and all objects in the next frame is the smallest among adjacent frames.

[0116] For example, refer to Figure 5The distance 1 between the object A1 in the previous frame picture and the object A2 in the next frame picture is the smallest among the distances 1, 2 and 3, and thus the object A1 in the previous frame picture and the object A2 in the next frame picture satisfy the first preset condition. The distance 4 between the object B1 in the previous frame picture and the object B2 in the next frame picture is the smallest among the distances 4, 5 and 6, and thus the object B1 in the previous frame picture and the object B2 in the next frame picture satisfy the first preset condition.

[0117] (3) If the centroid distance satisfies the first preset condition, the objects between the adjacent frame pictures are determined as matched objects, and the matched objects are obtained.

[0118] It can be understood that the position movement of each object in the device caused by the noise generated by the vibration in the normal operation process is generally not too large, and thus the distance between the same object in the adjacent frame pictures is generally smaller than the distance between different objects in the adjacent frame pictures. Therefore, in the embodiment, if the centroid distance between a certain object in the previous frame picture in the adjacent frame pictures and each object in the next frame picture is the smallest, the objects between the adjacent frame pictures are determined as matched objects.

[0119] For example, Figure 5 In the embodiment, the distance between the object A1 in the previous frame picture in the adjacent frame pictures and the object A2 in the next frame picture is the smallest, and thus the object A2 is the matched object of the object A1, and the distance between the object B1 in the previous frame picture in the adjacent frame pictures and the object B2 in the next frame picture is the smallest, and thus the object B2 is the matched object of the object B1.

[0120] S3, generating a centroid tracking trajectory of the matched object according to the centroid positions of the matched object in time sequence, the centroid tracking trajectory of the matched object being a tracking trajectory of the centroid of the matched object in the continuous frame pictures.

[0121] The continuous frame pictures can be a section of the video or the entire video, and the continuous picture sequence obtained by the frame processing.

[0122] For example, the tracking trajectory of the centroid of the matched object A in the continuous n frame pictures can be represented as A={A1, A2, A3, A4, …, An}, where n is the order of the plurality of continuous frame pictures, A1 is the centroid position of the object A in the first frame picture in the video, and An is the centroid position of the object A in the nth frame picture in the video.

[0123] Based on the above embodiment, in an embodiment, determining the target object in which the noise occurs in the device, the centroid position of the target object and the target time point in which the noise occurs according to the centroid tracking trajectory can include:

[0124] S4, obtain the center-of-mass distance of the matched object between each adjacent frame picture from the center-of-mass tracking trajectory of the matched object.

[0125] For example, the first center-of-mass distance between the matched object A1 and the matched object A2, the second center-of-mass distance between the matched object A2 and the matched object A3, and the n-1th center-of-mass distance between the matched object An-1 and the matched object An in the center-of-mass tracking trajectory A of the object A are obtained.

[0126] S5, if the center-of-mass distance of the matched object between adjacent frame pictures meets a second preset condition, the matched object is determined as a target object that appears noise, and the center-of-mass position of the target object is obtained.

[0127] The second preset condition is that the center-of-mass distance of the matched object between adjacent frame pictures is greater than a preset value. The preset value can be set according to the actual running condition of the intelligent device. The center-of-mass position of the target object can be the average value of the center-of-mass positions of the target object in each frame picture obtained from the video, or the center-of-mass position in the picture corresponding to the time when the target object appears noise.

[0128] It can be understood that if the center-of-mass distance between adjacent frame pictures is greater than the preset value, it means that the vibration amplitude of the object is too large, the noise emitted will exceed the threshold, and it may affect the user experience. Therefore, at this time, the object can be determined as a target object, so that noise processing is performed on the target object to reduce the target noise of the target object.

[0129] S6, determine the time point corresponding to the latter frame picture in the adjacent frame pictures as a target time point when the noise appears.

[0130] For example, after obtaining the first center-of-mass distance between A1 and A2, the second center-of-mass distance between A2 and A3, and the n-1th center-of-mass distance between An-1 and An in the center-of-mass tracking trajectory A of the object A, the n-1 center-of-mass distances between the n frame pictures of the object A are compared with the preset value respectively. If the n-1th center-of-mass distance, that is, the vibration amplitude between the object A in the n-1th frame picture of the video and the object A in the nth frame picture is too large, the object A can be determined as a target object that appears noise, the nth frame is the target time point when the object A appears noise, and the center-of-mass position of the object An in the nth frame picture can be the center-of-mass position of the target object.

[0131] Based on the above embodiment, optionally, the object can include a new object, and the center-of-mass tracking is performed on at least one object according to the video to generate a center-of-mass tracking trajectory of each object, and can further include:

[0132] It should be noted that the new object can be an object detected in a frame other than the first frame of the video. The camera can not detect the new object in the previous frame during the running of the device, or the new object can not run in the previous frame of the video, no position movement occurs, or a fault occurs in the device to cause the object to be added in the video. Therefore, when the camera collects each object running in the device, the new object is not detected at the beginning, and the new object is detected in the subsequent frame. For example Figure 5 In the example, the objects A1 and B1 are objects detected in the first frame of the video, and the object C2 is added in the second frame, so the object C2 is a new object.

[0133] S205, obtaining the number of objects in the adjacent frame pictures.

[0134] For example, referring to Figure 5 , the objects in the first frame picture of the video include objects A1 and B1, so the number of objects in the first frame picture is 2, and the objects in the second frame picture include objects A2, B2 and C2, so the number of objects in the first frame picture is 3.

[0135] S206, if the number of objects in the latter frame picture is greater than the number of objects in the former frame picture, and the centroid distance of the objects between the adjacent frame pictures does not satisfy the first preset condition, it is determined that there is a new object in the latter frame picture.

[0136] For example, Figure 5 In the example, the number of objects in the second frame picture is greater than the number of objects in the first frame picture, and there is no object in the first frame that matches the object C2 in the second frame, so it is determined that there is a new object C2 in the second frame picture.

[0137] S207, obtaining the centroid position of the new object.

[0138] S208, generating a centroid tracking trajectory of the new object with the centroid position of the new object as an initial position.

[0139] For example, Figure 5 In the example, the identification information of the new object C2 is registered as object C, and the centroid tracking trajectory of the new object C is generated with the centroid position of the object C2 in the second frame picture as an initial position, that is, C = {C2, C3, …, Cm-1, Cm}, where m in the centroid tracking trajectory of object C and n in the centroid tracking trajectory of object A can be the same or different. If they are the same, it means that objects A and C both exist in the video at the nth frame. If n is less than m, it means that object B no longer appears in the video after the nth frame, and object C still appears in the video at the mth frame.

[0140] Optionally, the object can include a disappeared object, and after the center-of-mass tracking of the at least one object is performed according to the video to generate a center-of-mass tracking trajectory of each object, the method can further include:

[0141] It should be noted that the disappeared object can refer to an object that appears in the video but does not appear in subsequent frame pictures after a certain frame picture. For example, if object D is detected in the first frame of the video, but the center-of-mass position of the object D does not change during the subsequent running process, that is, the object D is stationary during the running process, so the object D generally does not have noise, and thus the center-of-mass tracking trajectory of the object D can be deleted to reduce the center-of-mass tracking calculation amount of each object in the intelligent device and improve the center-of-mass tracking calculation efficiency of each object in the intelligent device.

[0142] S209, if the object in the picture does not appear in the subsequent frame picture, it is determined that the object is a disappeared object;

[0143] S210, deleting the center-of-mass tracking trajectory of the disappeared object.

[0144] In the technical solution provided in the embodiment, the intelligent device performs center-of-mass tracking on each object in the running, determines a target object with noise based on the center-of-mass tracking trajectory of each object, and determines a target time point with noise and a position of the target object. Thus, the target object with noise is accurately positioned, and the accuracy of positioning the target noise is improved.

[0145] Based on the above embodiment, please refer to Figure 3 , Figure 3 Another flowchart of the noise processing method provided in the embodiment is shown. In an embodiment, the noise directional processing of the target noise can include the following steps according to the noise information, the center-of-mass position, and the target time point:

[0146] S301, determining a noise direction of the target noise according to the center-of-mass position.

[0147] The noise direction can be a direction of the center-of-mass position of the target object in the intelligent device relative to the position of the microphone.

[0148] For example, a preset microphone position coordinate is obtained, and a center-of-mass position coordinate of the target object is obtained, and the direction of the center-of-mass position of the target object relative to the position of the microphone is calculated according to the microphone position coordinate and the center-of-mass position coordinate of the target object. Please refer to Figure 6 , Figure 6Fig. 1 is a schematic diagram of processing target noise in a smart device, wherein in the embodiment, it can be assumed that the target noise is a uniform linear array, y1(k), y2(k) to yn(k) are target noises emitted by n array elements in the uniform linear array corresponding to the target noise, the higher the frequency or the greater the amplitude of the target noise, the more the number of array elements in the uniform linear array of the target noise and the greater the array element density. S(k) can be a noise processing signal, which can be emitted by a microphone in the smart device into the array elements of the target noise to cancel the target noise, for example, noise processing signal x1(k) is emitted into array element 1 in the target noise to cancel target noise y1(k), noise processing signal x2(k) is emitted into array element 2 in the target noise to cancel target noise y2(k), and the angle of the noise processing signal relative to the array element of the target noise can be the direction of the target noise relative to the microphone, i.e. the noise direction.

[0149] S302, generating a noise processing signal according to the noise direction and the target noise.

[0150] Optionally, generating a noise processing signal according to the noise direction and the target noise signal can include:

[0151] S1, obtaining noise information of the target noise.

[0152] The noise information of the target noise can include at least one of the number of array elements, the array element spacing, the sound speed, the noise signal frequency, and the target time point at which the target noise occurs.

[0153] For example, after the microphone provided in the smart device collects the target noise of the target object, the noise information of the target noise is analyzed to obtain at least one of the number of array elements, the array element spacing, the sound speed, and the noise signal frequency of the target noise.

[0154] S2, determining signal parameters of the noise processing signal according to the noise information of the target noise.

[0155] The signal parameters can include at least one of the emission angle of the noise processing signal, the signal frequency, and the target emission time point.

[0156] It can be understood that the noise processing signal generated by using the signal parameters can be used to cancel the target noise. Therefore, the signal phase of the noise processing signal and the signal phase of the target noise can be opposite or differ by 180°, for example, if the target noise is a sine wave, the noise processing signal can be a cosine wave. For example, the smart device calculates at least one of the emission angle of the noise processing signal, the signal frequency, and the target emission time point according to at least one of the number of array elements, the array element spacing, the sound speed, and the noise signal frequency of the target noise.

[0157] Optionally, since the smart device has a processing time difference in analyzing and processing the target noise to generate the noise processing signal or even emitting the noise processing signal, and there is no noise processing signal to process the target noise in the processing time difference, in this embodiment, the position and the reference time point of the reference noise of the smart device can be estimated according to the running information preset by the smart device, and the preset reference noise processing signal is obtained according to the position and the reference time point of the reference noise, which is used to compensate for the processing time difference of the noise processing signal. The reference noise can be the same as the target noise, or can be slightly different from the target noise.

[0158] For example, if the smart device is a washing machine, the running information can be a washing algorithm set in the washing machine, such as water inlet time, rinsing time, dehydration time, drying time, etc. The reference time point of the noise is calculated according to at least one of the water inlet time, the rinsing time, the dehydration time, and the drying time of the washing machine, and the preset reference noise processing signal is obtained at the reference time point and emitted to the position of the reference noise to eliminate the reference noise.

[0159] S3, generating a noise processing signal according to the signal parameter.

[0160] For example, the noise processing signal can be generated by a three-dimensional beamforming method, that is, after the noise processing signal is generated, the transmission path of the noise processing signal in space is tracked, so that the noise processing signal is sent only in the noise direction of the target noise at the target time point.

[0161] S303, performing noise directional processing on the target noise at the target time point according to the noise processing signal.

[0162] For example, by reasonably setting the amplitude and phase of the noise processing signal emitted by each array element in the microphone, the signal beam is shot to the noise direction of the target noise at the target time point by the joint action of each array element in the microphone, so as to realize the noise directional processing on the target noise.

[0163] In the technical scheme provided in this embodiment, the noise direction of the target noise is determined according to the centroid position, the noise processing signal is generated according to the noise direction and the target noise, and the noise directional processing is performed on the target noise at the target time point according to the noise processing signal. In this way, by reasonably setting the amplitude and phase of the noise processing signal emitted by each array element in the microphone, the signal beam is shot to the noise direction of the target noise at the target time point by the joint action of each array element in the microphone, so as to enhance the intensity of the noise processing signal in the noise direction, and improve the effect of noise processing of the smart device.

[0164] To facilitate better implementation of the noise processing method provided in the embodiments of the present application, the embodiments of the present application further provide a device based on the above noise processing method. The meanings of the terms are the same as those in the above noise processing method, and the specific implementation details can be referred to the description in the method embodiments.

[0165] Please refer to Figure 7 , Figure 7 To facilitate better implementation of the noise processing method provided in the embodiments of the present application, the embodiments of the present application further provide a device based on the above noise processing method. The meanings of the terms are the same as those in the above noise processing method, and the specific implementation details can be referred to the description in the method embodiments.

[0166] The first acquisition module 301 is configured to acquire a video of running of at least one object in the device.

[0167] The generation module 302 is configured to perform centroid tracking on the at least one object according to the video, to generate a centroid tracking trajectory of each object.

[0168] The determination module 303 is configured to determine, according to the centroid tracking trajectory, a target object in which noise occurs in the device, and a centroid position of the target object and a target time point at which the target object occurs noise.

[0169] The second acquisition module 304 is configured to acquire, at the target time point, a target noise emitted by the target object and noise information of the target noise.

[0170] The processing module 305 is configured to perform noise directional processing on the target noise according to the noise information, the centroid position and the target time point.

[0171] Optionally, the noise processing device 300 can further include:

[0172] The frame dividing module is configured to perform frame dividing processing on the video to obtain a plurality of picture sequences.

[0173] The arrangement module is configured to arrange the plurality of picture sequences into a plurality of continuous frame pictures in time sequence.

[0174] The first determination module is configured to perform object detection on each picture in the plurality of continuous frame pictures, to determine a centroid position of at least one object in each picture.

[0175] The first generation module is configured to perform trajectory tracking on the at least one object according to the centroid position in the order of the plurality of continuous frame pictures, to generate a tracking trajectory of the centroid of the at least one object in the plurality of continuous frame pictures.

[0176] The first acquisition module is configured to acquire the centroid position of the at least one object in each picture.

[0177] The matching module is configured to match each object according to the centroid position of each object in the adjacent frame pictures to determine matched objects, and obtain the matched objects.

[0178] The second generation module is configured to generate a centroid tracking trajectory of the matched objects in a time sequence according to the centroid positions of the matched objects, the centroid tracking trajectory of the matched objects being a tracking trajectory of the centroid of the matched objects in the continuous frame pictures.

[0179] The second acquisition module is configured to acquire the centroid distance of each object between each adjacent frame picture.

[0180] The judgment module is configured to judge whether the centroid distance meets a first preset condition.

[0181] The first determination module is configured to determine that the objects between the adjacent frame pictures are matched objects to obtain the matched objects if the centroid distance meets the first preset condition.

[0182] The third acquisition module is configured to acquire the centroid distance of the matched objects between each adjacent frame picture from the centroid tracking trajectory of the matched objects.

[0183] The second determination module is configured to determine that the object is a target object with noise and acquire the centroid position of the target object if the centroid distance of the matched objects between the adjacent frame pictures meets a second preset condition.

[0184] The third determination module is configured to determine that a time point corresponding to a later frame picture in the adjacent frame pictures is a target time point with noise.

[0185] The fourth acquisition module is configured to acquire the number of objects in the adjacent frame pictures.

[0186] The second determination module is configured to determine that there is a new object in the later frame picture if the number of objects in the later frame picture is greater than the number of objects in an earlier frame picture and the centroid distance of the objects between the adjacent frame pictures does not meet the first preset condition.

[0187] The fifth acquisition module is configured to acquire the centroid position of the new object.

[0188] The third generation module is configured to generate a centroid tracking trajectory of the new object with the centroid position of the new object as an initial position.

[0189] The third determination module is configured to determine that the object is a disappeared object if the object in the frame picture does not appear in a matched object in a subsequent frame picture.

[0190] The deletion module is configured to delete the centroid tracking trajectory of the disappeared object.

[0191] The detection module is configured to detect contour information of at least one object in each frame of picture.

[0192] The fourth determination module is configured to determine a bounding box of the at least one object according to the contour information.

[0193] The fifth determination module is configured to determine a centroid position of the at least one object according to the bounding box.

[0194] The sixth determination module is configured to determine a noise direction of the target noise according to the centroid position.

[0195] The fourth generation module is configured to generate a noise processing signal according to the noise direction and the target noise.

[0196] The processing module is configured to perform noise directional processing on the target noise at a target time point according to the noise processing signal.

[0197] The sixth acquisition module is configured to acquire noise information of the target noise.

[0198] The seventh determination module is configured to determine a signal parameter of the noise processing signal according to the noise information of the target noise.

[0199] The fifth generation module is configured to generate the noise processing signal according to the signal parameter.

[0200] Embodiments of the present application also provide a kind of intelligent device, the intelligent device can be mobile phone and the like terminal, as shown in Fig. Figure 8 It shows the structure schematic diagram of the intelligent device related to the embodiments of the present application, specifically to:

[0201] The intelligent device can include one or more than one processor 401 of processing core, one or more than one computer readable storage medium of memory 402, power supply 403 and input unit 404 and the like components.The person skilled in the art can understand, Figure 8 The structure of the intelligent device shown in the figure does not constitute the limitation to the intelligent device, can include more or less than the components shown in the figure, or combine certain components, or different component arrangement. Among them:

[0202] The processor 401 is the control center of the smart device, connects all parts of the smart device through various interfaces and lines, executes various functions of the smart device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the smart device as a whole. Optionally, the processor 401 can include one or more processing cores; preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0203] The memory 402 can be used to store software programs and modules, and the processor 401 executes various functions and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the smart device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access for the processor 401 to the memory 402.

[0204] The smart device further includes a power supply 403 for supplying power to various components, and preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging and power consumption management through the power management system. The power supply 403 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, etc.

[0205] The smart device can further include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0206] Although not shown, the smart device can also include a display unit, etc., which will not be described here. Specifically, in the present embodiment, the processor 401 in the smart device will load the executable file corresponding to the process of one or more than one application program into the memory 402 according to the following instructions, and run the application program stored in the memory 402 by the processor 401, thereby realizing various functions, as follows:

[0207] The video of the running of at least one object inside the device is acquired; the center-of-mass tracking of the at least one object is performed according to the video to generate a center-of-mass tracking trajectory of each object; a target object that generates noise in the device is determined according to the center-of-mass tracking trajectory, and a center-of-mass position of the target object and a target time point at which the target object generates noise are determined; target noise generated by the target object at the target time point is acquired, and noise information of the target noise is acquired; and the target noise is subjected to noise directional processing according to the noise information, the center-of-mass position, and the target time point. In this way, the intelligent device can determine the center-of-mass position of the target object that generates noise, the target noise, and the target time point at which the target object generates noise according to the center-of-mass tracking trajectory of the object when the object inside the device generates noise, so as to perform noise directional processing on the target noise at the target time point, thereby accurately performing active noise reduction processing on the noise and improving the accuracy of noise processing.

[0208] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the detailed description of the noise processing method above, which will not be described here again.

[0209] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the intelligent device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the intelligent device to perform the method provided in various optional implementation manners in the above embodiments.

[0210] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by computer instructions, or by relevant hardware controlled by computer instructions, which can be stored in a storage medium and loaded and executed by a processor, and the storage medium is a computer readable storage medium. Therefore, the embodiments of the present application provide a storage medium in which a computer program is stored, and the computer program can include computer instructions, which can be loaded by a processor to execute any noise processing method provided by the embodiments of the present application.

[0211] The specific implementation of each operation can refer to the previous embodiments, which will not be described here again.

[0212] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0213] Since the computer instructions stored in the storage medium can execute any one of the noise processing methods provided by the embodiments of the present application, the beneficial effects that can be achieved by any one of the noise processing methods provided by the embodiments of the present application can be achieved. Details are described above, and will not be repeated here.

[0214] The above describes in detail a noise processing method, device, intelligent device and storage medium provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A noise processing method, characterized in that, The method includes: Acquire video of at least one object running inside the device; Centroid tracking is performed on the at least one object based on the video to generate a centroid tracking trajectory for each object; The target object in the device that generates noise is determined based on the centroid tracking trajectory, as well as the centroid position of the target object and the target time point when the noise occurs; wherein, the target object is the object that generates noise due to a large movement of its centroid position in the centroid tracking trajectory; the target time point is the time point of the video frame corresponding to the large movement of the centroid position of the target object; At the target time point, acquire the target noise emitted by the target object and the noise information of the target noise; Based on the noise information, centroid location, and target time point, noise direction processing is performed on the target noise.

2. The noise treatment method according to claim 1, characterized in that, The step of performing centroid tracking on the at least one object based on the video to generate a centroid tracking trajectory for each object includes: The video is segmented into frames to obtain a multi-frame image sequence; The multi-frame image sequence is arranged into multiple consecutive frame images in chronological order; Object detection is performed on each of multiple consecutive frame images to determine the centroid position of at least one object in each frame image; Based on the centroid position, the at least one object is tracked in the order of the plurality of consecutive frame images to generate the tracking trajectory of the centroid of the at least one object in the plurality of consecutive frame images.

3. The noise treatment method according to claim 2, characterized in that, The object includes a matching object. The step of tracking the trajectory of the at least one object according to the centroid position in the order of the plurality of consecutive frame images, generating a tracking trajectory of the centroid of the at least one object in the plurality of consecutive frame images, includes: Obtain the centroid position of at least one object in each frame of the image; Each object is matched based on the centroid position of each object in adjacent frame images to determine the matching object; The centroid positions of the matched objects are arranged in chronological order to generate centroid tracking trajectories for the matched objects. The centroid tracking trajectory of the matched objects is the tracking trajectory of the centroid of the matched objects in the consecutive frame images.

4. The noise treatment method according to claim 3, characterized in that, The step of matching each object based on the centroid position of each object in adjacent frame images to determine the matched object and obtain the matched object includes: Get the centroid distance of each object between adjacent frames; Determine whether the centroid distance satisfies the first preset condition; If the centroid distance satisfies the first preset condition, then the object between the adjacent frame images is determined to be a matching object, and the matching object is obtained.

5. The noise treatment method according to claim 4, characterized in that, The step of determining the target object in the device where noise occurs, the centroid position of the target object, and the target time point where noise occurs based on the centroid tracking trajectory includes: From the centroid tracking trajectory of the matched object, obtain the centroid distance between the matched objects in each adjacent frame image; If the centroid distance between the matched objects in adjacent frame images meets the second preset condition, then the matched object is determined to be a target object with noise, and the centroid position of the target object is obtained; the second preset condition is that the centroid distance between the matched objects in adjacent frame images is greater than a preset value; The time point corresponding to the next frame in the adjacent frame images is determined as the target time point where noise occurs.

6. The noise treatment method according to claim 4, characterized in that, The objects include newly added objects, and the step of performing centroid tracking on the at least one object based on the video to generate a centroid tracking trajectory for each object includes: Get the number of objects in adjacent frame images; If the number of objects in the next frame is greater than the number of objects in the previous frame, and the centroid distance between the objects in the adjacent frames does not meet the first preset condition, then it is determined that there are newly added objects in the next frame. Obtain the centroid position of the newly added object; Using the centroid position of the newly added object as the initial position, generate the centroid tracking trajectory of the newly added object.

7. The noise treatment method according to claim 4, characterized in that, The objects include vanishing objects. After performing centroid tracking on at least one object based on the video to generate a centroid tracking trajectory for each object, the process includes: If the object in the image does not appear as a matching object in subsequent frames, then the object is determined to be a missing object. Delete the centroid tracking trajectory of the disappeared object.

8. The noise treatment method according to claim 2, characterized in that, The step of performing object detection on each of multiple consecutive image frames to determine the centroid position of at least one object in each image frame includes: Detect the contour information of at least one object in each frame of the image; Determine the bounding box of the at least one object based on the contour information; The centroid position of the at least one object is determined based on the bounding box.

9. The noise processing method according to claim 1, characterized in that, The step of performing noise direction processing on the target noise based on the noise information, centroid position, and target time point includes: The noise direction of the target noise is determined based on the centroid position; Based on the noise direction and the target noise, a noise processing signal is generated; Based on the noise processing signal, noise direction processing is performed on the target noise at the target time point.

10. The noise processing method according to claim 9, characterized in that, The step of generating a noise processing signal based on the noise direction and the target noise signal includes: Obtain noise information of the target noise; Based on the noise information, determine the signal parameters of the noise processing signal; The noise processing signal is generated based on the signal parameters.

11. A noise treatment device, characterized in that, include: The first acquisition module is used to acquire video of at least one object running inside the device; A generation module is used to perform centroid tracking on the at least one object based on the video to generate a centroid tracking trajectory for each object; The determination module is used to determine, based on the centroid tracking trajectory, the target object in the device that generates noise, the centroid position of the target object, and the target time point where the noise occurs; wherein, the target object is the object that generates noise due to a large movement of its centroid position in the centroid tracking trajectory; and the target time point is the time point of the video frame corresponding to the large movement of the centroid position of the target object. The second acquisition module is used to acquire the target noise emitted by the target object and the noise information of the target noise at the target time point; The processing module is used to perform noise direction processing on the target noise based on the noise information, the centroid position, and the target time point.

12. A smart device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the noise processing method as described in any one of claims 1 to 10 when it invokes the computer program in the memory.

13. A storage medium, characterized in that, The storage medium is used to store a computer program, which is loaded by a processor to execute the noise processing method according to any one of claims 1 to 10.

Images