A household unmanned control management system based on laundry robot

Abstract

The present invention belongs to the technical field of machine control management, and specifically discloses a household unmanned control management system based on a laundry robot, including a laundry robot body and a laundry control system, wherein the laundry control system includes a clothing recognition and analysis module, an environmental information collection module, a clothing washing confirmation module, a user interaction verification terminal, a database, and a clothing processing execution module. When collecting clothing information of the dirty clothing placement area, the present invention determines whether the dirty clothing placement area needs layered detection based on the collected overall scanning visual image, overall spectral data, and overall scanning depth image, which effectively solves the current problem of insufficient attention to the accuracy of clothing recognition, and facilitates the subsequent automatic washing mode, water volume, and detergent addition amount of the clothing to be accurately controlled, reducing misjudgment, and at the same time, by confirming the washing operation instructions, the flexibility and pertinence of the washing settings are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine control management, and in particular, relates to an unmanned household control management system based on a laundry robot. Background Art

[0002] As a daily chore, traditional manual or semi-automatic laundry methods are time-consuming and labor-intensive, and cannot meet the needs of people's fast-paced lives. Therefore, people are eager for a more convenient and efficient laundry solution. The emergence of laundry robots just meets this demand. In order to ensure the user experience, unmanned control and management are required.

[0003] Prior art, such as the smart home laundry control system disclosed in the Chinese invention patent application with application number 201711140833.8, includes a main body, the outside of the main body is provided with a rubber shockproof sleeve, the inside of the main body is provided with a temperature control device, a human-machine interaction component, a display component, a signal transceiver component and a water volume control component, and the temperature control device, display component, signal transceiver component and water volume control component are all electrically connected to the human-machine interaction component. The temperature control device, human-machine interaction component, display component, signal transceiver component and water volume control component are all provided with a single-chip microcomputer, which can make people's life more convenient and the home environment more intelligent.

[0004] Prior art, such as a washing machine control method and a laundry system disclosed in the Chinese invention patent application with application number 201710725577.2, wherein the washing machine is connected to a smart terminal for communication, and the washing machine has a home working mode and an out-of-home working mode. When the washing machine is switched to the out-of-home working mode, the washing machine keeps the door locked after the washing process is completed and waits for the smart terminal to issue a control instruction. When the washing machine is switched to the home working mode, the washing machine unlocks the door after the washing process is completed. Thus, a more reasonable and safe design can be made for the field of remote control of washing machines by smart terminals. When the washing machine is in the out-of-home working mode, the subsequent operations of the washing machine after the washing process is completed are controlled to prevent wrinkles in the clothes, which is convenient for users to use. At the same time, it also prevents the washing machine from being directly unlocked after the washing process is completed, causing the remote user to be unable to further control the washing machine after the washing process is completed. At the same time, the washing machine also automatically switches the working mode according to the distance between the smart terminal and the machine, which improves the intelligence level of the washing machine.

[0005] Another example of the prior art is a control method for a smart home system disclosed in a Chinese invention patent application with application number 202110585909.8, which includes obtaining the user's clothing information. Based on the clothing information, the user is interacted with to confirm whether there is a need for laundry. By obtaining the user's clothing information and interacting with the user to confirm whether there is a need for laundry, the user can be intelligently reminded to wash the clothes, avoiding the user from forgetting and preventing the accumulation of clothes, which can improve the intelligence of the home system and thus improve the user's experience.

[0006] With regard to the above technical solutions, it is obvious that the current management of laundry robots still has the following deficiencies: 1. Insufficient attention is paid to the accuracy of clothing recognition, and the various characteristics of clothing cannot be accurately grasped, which makes it impossible to accurately control the automatic washing mode, water volume and detergent addition amount of clothing, thereby failing to guarantee the washing effect.

[0007] 2. There is a lack of perception of the external environment, which has not been integrated into the laundry control process, and thus no adaptive adjustment has been made, resulting in a lack of flexibility in laundry control and difficulty in ensuring the stability of washing quality.

[0008] 3. The current human-computer interaction relies more on the interactive signals given by the user to realize functions such as reminders, which is difficult to fully meet the personalized needs of users. Cleaning instruction analysis is not performed, and the flexibility and pertinence of cleaning settings cannot be guaranteed, and resource utilization cannot be improved. Summary of the invention

[0009] In view of this, in order to solve the problems raised in the above background technology, a household unmanned control management system based on a laundry robot is proposed.

[0010] The objective of the present invention can be achieved through the following technical solutions: The present invention provides a household unmanned control management system based on a laundry robot, comprising a laundry robot main body and a laundry control system, the laundry control system comprising: a clothing recognition and analysis module, which is used to move the laundry robot to the dirty clothing placement area after receiving a washing instruction, and collect clothing information in the dirty clothing placement area.

[0011] The environmental information collection module is used to start the sensor terminal carried by the washing machine to collect environmental information.

[0012] The clothing washing confirmation module is used to confirm the washing operation instructions based on the environmental information.

[0013] The user interaction verification terminal is used to send the cleaning operation instruction to the user for confirmation and obtain the confirmed cleaning operation instruction.

[0014] The database is used to store the user's current accumulated images of each piece of clothing, store the clothing material spectrum data table, and store the reference clothing thickness of each material type, the acceptable washing water temperature range and the allowed mixed clear material type set.

[0015] The clothing processing execution module is used to control the water inlet valve of the washing machine to open according to the cleaning operation instructions, and perform corresponding cleaning, and dry the clothes after the cleaning is completed.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) When collecting clothing information of the dirty clothing placement area, the present invention determines whether the dirty clothing placement area needs layered detection based on the collected overall scanning visual image, overall spectral data and overall scanning depth image, thereby effectively solving the current problem of insufficient attention to the accuracy of clothing recognition, facilitating accurate grasp of various characteristics of clothing, and the subsequent automatic washing mode, water volume and detergent addition amount of the clothing can all be accurately controlled, thereby reducing misjudgment, and providing a strong guarantee for the subsequent washing effect.

[0017] (2) The present invention effectively compensates for the current lack of external environment perception by combining environmental information when confirming washing operation instructions. It fully integrates environmental information into the laundry control process and performs adaptive adjustments, thereby improving the flexibility of automated laundry control and ensuring the stability of washing quality.

[0018] (3) The present invention confirms the cleaning operation instruction and sends it to the user for confirmation, thereby making up for the deficiency that the current human-computer interaction relies more on the interactive signal given by the user to realize the reminder and other functions and is difficult to fully meet the personalized needs of the user. The personalized needs of the user can be fully met.

[0019] (4) When confirming the washing operation instructions, the present invention further analyzes the washing instructions by confirming the washing type, the clothing washing water temperature, and the target detergent addition amount under the corresponding washing type, thereby ensuring the flexibility and pertinence of the washing settings, while also improving the washing efficiency and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of the overall system structure of the present invention.

[0022] Figure 2This is a schematic diagram of the connection of the laundry control system modules of the present invention.

[0023] Figure 3 It is a schematic flow chart of the overall implementation steps of the present invention.

[0024] Figure 4 It is a schematic diagram of the process of collecting clothing information in the dirty clothing placement area of ​​the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1 As shown, the present invention provides a household unmanned control management system based on a laundry robot, which includes a laundry robot body and a laundry control system.

[0027] See also Figure 2 and Figure 3 As shown, the laundry control system includes a clothing recognition and analysis module, an environmental information collection module, a clothing cleaning confirmation module, a user interaction verification terminal, a database and a clothing processing execution module.

[0028] In the above, the clothing identification and analysis module is connected to the clothing cleaning confirmation module and the database respectively, the clothing cleaning confirmation module is connected to the environment information collection module, the database and the user interaction verification terminal respectively, and the user interaction verification terminal is also connected to the clothing processing execution module.

[0029] The clothing recognition and analysis module is used to move to the dirty clothing placement area after the laundry robot receives a washing instruction and collect clothing information in the dirty clothing placement area.

[0030] Understandably, the laundry robot has an advanced navigation system built in, such as integrating laser radar, visual recognition, and inertial navigation technologies, so that after receiving the washing command, the laundry robot can autonomously move to the clothing storage area, where the clothing storage area can be the dirty clothes basket and the designated clothing storage area, etc., and can effectively avoid various obstacles in the home during the movement process to ensure smooth arrival at the target location. Among them, mobile obstacle avoidance is a relatively mature technology, and the obstacle avoidance of the laundry robot is not the main consideration of the present invention, so how to avoid obstacles and how to plan the path when it autonomously moves to the clothing storage area will not be described in detail here.

[0031] Specifically, Figure 4As shown, collecting clothing information in the dirty clothing placement area includes: S1, starting a high-definition camera placed in the dirty clothing placement area to perform an overall scan to obtain an overall scan visual image.

[0032] S2. Start the multi-spectral sensor carried by the laundry robot to perform an overall scan of the dirty clothes placement area to obtain overall spectral data of the scan.

[0033] S3, start the depth sensor carried by the laundry robot to perform an overall scan of the dirty clothes placement area to obtain an overall scan depth image.

[0034] S4. Determine whether the dirty clothes placement area requires layered detection.

[0035] S5. If the answer is yes, confirm the number of layers, and start the mechanical arm of the laundry robot according to the number of layers to gently spread the clothes and layer them in sequence.

[0036] S6. When each layer of clothing is exposed, it is scanned by the high-definition camera, multi-spectral sensor and depth sensor carried by the laundry robot, and the scanned visual image, spectral data and scanned depth image of each layer are used as the clothing information of the dirty clothing placement area.

[0037] S7. If the judgment is no, the overall scanned visual image, the overall spectral data and the overall scanned depth image are used as the clothing information of the dirty clothing placement area.

[0038] It should be added that before starting the multi-spectral sensor on the laundry robot to scan the dirty clothes placement area as a whole, it is necessary to reasonably set the measurement parameters according to the clothing material identification requirements and the performance characteristics of the spectrometer used. This includes parameters such as spectral resolution, integration time, scanning range, etc., to ensure the collection of high-quality spectral data.

[0039] Understandably, by selecting appropriate settings such as the slit width, the spectrometer can distinguish spectral features with smaller wavelength intervals. Generally, for clothing material identification, a spectral resolution of several nanometers can meet the basic requirements. The integration time is determined based on factors such as ambient light intensity and clothing reflectivity to ensure that the collected spectral signal has sufficient intensity and signal-to-noise ratio. For example, in a normal indoor lighting environment, the integration time can be set between tens of milliseconds and hundreds of milliseconds. The scanning range is determined based on the key spectral bands to be detected and the material identification features that may be involved. For example, if the focus is on the visible light to near-infrared band, the scanning range can be set to 400-2500nm.

[0040] It should be added that the collected spectral data may be interfered by factors such as ambient light fluctuations and the instrument's own electronic noise, generating some noise signals and occasional abnormal values, such as suddenly appearing too high or too low spectral intensity values. Digital signal processing technology is used to remove these interference factors. Common methods include smoothing filtering, such as moving average filtering, which replaces the original data points by calculating the average value of the spectral intensity data in a certain window, making the spectral curve smoother and removing high-frequency noise. There are also algorithms such as median filtering, which replaces the original data points with the median value of the data in the sorted window, which is better for removing abnormal values ​​such as impulse noise, etc., to improve the quality and stability of spectral data.

[0041] In a specific embodiment, due to the influence of background light and instrument baseline drift during the spectrum measurement process, the baseline of the spectrum curve, that is, the theoretically stable intensity reference line that should be presented when there is no sample absorption or reflection, is offset or bent, affecting the judgment of key information such as material characteristic peaks. Baseline correction methods such as polynomial fitting baseline correction and piecewise linear fitting baseline correction are used to restore the baseline of the spectrum curve to an ideal horizontal state, highlighting the spectral absorption and reflection characteristics of the clothing material itself.

[0042] It can be understood that the polynomial fitting baseline correction is to fit the baseline with a polynomial function according to the overall trend of the spectral curve, and then subtract the fitted baseline data from the original spectral data to obtain the corrected spectral curve. The piecewise linear fitting baseline correction is to divide the spectral curve into certain wavelength intervals, and perform linear fitting and correction on the baseline in each interval.

[0043] In a specific embodiment, spectral characteristic parameters are used to describe the spectral characteristics of clothing. Common spectral characteristic parameters include the position of the absorption peak or reflection peak, peak height, half-peak width, peak area, etc. In addition, shape characteristic parameters such as the slope and curvature of the spectral curve within a certain wavelength range can also be calculated. The combination of these parameters can more comprehensively characterize the spectral characteristics of clothing and provide a rich basis for material identification.

[0044] It can be understood that the position of the absorption peak or reflection peak, that is, the corresponding wavelength value, is determined by finding the extreme points in the spectral curve. These peak positions of different materials are usually different and can be used as an important basis for identification. The peak height, that is, the difference between the spectral intensity corresponding to the peak point and the baseline intensity, reflects the degree of absorption or reflection of the material to light of a specific wavelength. The half-peak width, that is, the width of the absorption peak or reflection peak at half height, reflects the width of the characteristic peak, which is related to the internal structure of the material, molecular vibration and other characteristics. The peak area refers to the area enclosed by the absorption peak or reflection peak and the baseline, which comprehensively reflects the total absorption or reflection characteristics of the material in this wavelength region.

[0045] Furthermore, step S4 determines whether the dirty clothes placement area needs layered detection, specifically including: S41, using image recognition technology to identify the overall scanned visual image, outputting the three-dimensional contour of the overall scanned visual image, and extracting the contour of each piece of clothing therefrom.

[0046] S42. For each pair of adjacent clothing contours, calculate the shortest distance between them and compare them with the set reference distance. If the shortest distance between a pair of adjacent clothing is lower than the set reference distance, record the pair of adjacent clothing as a close fitting adjacent clothing pair. Count the number of close fitting adjacent clothing pairs and divide it by the number of adjacent clothing pairs to obtain a close fitting clothing ratio.

[0047] S43. Based on the outline of each piece of clothing, determine the area and area of ​​each area enclosed by the outline of each piece of clothing, calculate the overlapping area of ​​each adjacent clothing outline area, and divide it by the area of ​​the smaller clothing outline in each adjacent clothing outline area to obtain the overlapping ratio of each adjacent clothing outline area, and select the maximum value as the target clothing outline area overlapping ratio.

[0048] S44. Based on the overall scan depth image, confirm the overall posture layering requirement .

[0049] S45. Extract the spectral data table of clothing materials from the database, and then extract the spectral feature vector corresponding to each clothing material, extract each spectral feature parameter of each spectral curve from the overall spectral data, and combine them in a certain order to construct a spectral feature vector, and then obtain each material type based on a pre-set material identification algorithm, and count the number of material types.

[0050] S46. The ratio of tightly fitting clothing being greater than the set reference tightly fitting clothing ratio is defined as condition 1, the overlap ratio of the target clothing contour area being greater than the set reference clothing contour area overlap ratio is defined as condition 2, the overall posture stratification requirement being greater than the set value is defined as condition 3, and the number of material types being greater than the set reference material type number is defined as condition 4.

[0051] S47, determine whether any of the conditions 1, 2, 3 and 4 are satisfied. If so, it will be used as the judgment result. If not, it will not be used as the judgment result.

[0052] What needs to be added about step S42 is that for each pair of adjacent clothing contours, the shortest distance between them is calculated. This distance can be obtained by calculating the Euclidean distance between each point on the contour, and the set of corresponding points with the shortest distance between the two contours is found, and its distance value is the shortest distance between the two contours. For example, there is a top contour A and a trouser contour B. The shortest distance value between them is calculated to be d, where the distance formula can refer to the straight-line distance formula between two points in two-dimensional space and be expanded accordingly in three-dimensional space.

[0053] What needs to be added about step S42 is that the reference distance can be determined by analyzing and counting a large number of known fitted and non-fitted clothing images, or it can be estimated based on actual conditions such as the average thickness of the clothing. Generally speaking, if the average thickness of the clothing is around a few millimeters, the reference distance can be set to a value slightly larger than the average thickness of the clothing, such as 2 centimeters.

[0054] Understandably, the determination of the area enclosed by the outline of each piece of clothing in step S43 can be achieved by calculating the pixel point set inside the outline, etc. The calculation of the overlapping area and the area enclosed by the outline of each piece of clothing can be specifically performed by using an image analysis algorithm to determine the overlapping portion of two clothing outlines projected onto a two-dimensional plane in a three-dimensional space, counting the number of pixels contained in the overlapping portion, and then converting it into the actual overlapping area size according to information such as the resolution of the image.

[0055] The spectral feature vector in step S45 may include parameter values ​​such as the absorption peak position, peak height, half-peak width and slope of the spectral curve in several key characteristic wavelength regions. This facilitates the subsequent classification and identification of material components using algorithms such as pattern recognition in the form of feature vectors.

[0056] The material identification algorithm pre-set in step S45 can use a suitable pattern recognition or classification algorithm, such as identifying the material composition of unknown clothing based on the existing spectral feature vector and material annotation information. Common algorithms include distance-based classification methods, such as Euclidean distance, Mahalanobis distance, etc., which calculate the distance between the spectral feature vector of unknown clothing and the feature vectors of various known materials in the database. The closest material is the identification result. Specifically, a decision tree algorithm can be used, that is, a tree-shaped decision structure is constructed according to the spectral feature parameters, and the clothing material category is determined by judging the characteristic conditions layer by layer. Another specific example is a support vector machine, that is, by finding an optimal classification hyperplane, the spectral feature vectors of different materials are classified and distinguished in a high-dimensional space, as well as a classification model based on deep learning. According to the actual application scenario, data scale, and requirements for identification accuracy, one or more suitable algorithms are selected for material identification, wherein the classification model is specifically a convolutional neural network, which is trained through a large amount of spectral feature vector data, automatically learns the complex spectral feature patterns of different materials, and realizes accurate identification of unknown clothing materials.

[0057] In a specific embodiment, the image recognition technology is a relatively mature processing technology. The recognition algorithm principle and specific recognition process are not described here. By way of example, the reference tightly fitting clothing ratio can be set to a specific value of 0.5, and the reference clothing contour area overlap ratio can be set to a specific value of 0.75.

[0058] The step S44 of confirming the overall posture layering requirement includes: R1, based on the overall scanning depth image, obtaining the point cloud data of the clothing in the three-dimensional space, using a mathematical algorithm to calculate the minimum circumscribed sphere of the clothing point cloud data, and calculating the minimum circumscribed sphere volume, recorded as .

[0059] R2, use 3D reconstruction technology to construct the overall 3D posture contour of the clothing, and use the edge detection algorithm to extract the contour of the overall scanned depth image of the clothing, and then align and fuse it with the overall scanned visual image to obtain the fused 3D posture contour, and extract the volume and height of the fused 3D posture contour, which are recorded as and .

[0060] R3. Extract the reference clothing thickness of each material type from the database, and select the maximum reference thickness, which is recorded as .

[0061] R4. Calculate the overall posture layering requirement ,

[0062] , represents a natural constant, To set the minimum external sphere volume of the clothes pile for reference, is the reference stacking height to be set, To set the reference material thickness, To set the reference volume ratio.

[0063] Understandably, It represents the overlap ratio between the fused 3D posture contour volume and the minimum circumscribed sphere volume. The more overlapped, the more regular the posture is and the greater the probability of being covered by clothes. Therefore, the overlap ratio is used to correct the minimum circumscribed sphere volume. ,in, It represents the correction factor of material thickness on the height of 3D posture profile, that is, clothing thickness has a positive effect on the stacking height, and it is necessary to reduce the influence of clothing thickness on the analysis results.

[0064] It should be added that the edge detection algorithm can be used to extract the contour of clothing by using an edge detection method based on a depth gradient. The edge pixels are determined by calculating the rate of change of the depth value in different directions, and then the contour of the clothing is obtained based on the edge pixels.

[0065] It should be added that the minimum circumscribed sphere volume for reference of clothes stacking is an ideal state in which clothes are placed relatively loosely and basically without overlap, which is set artificially. The minimum circumscribed sphere volume is calculated in the same way in advance and saved as the reference minimum circumscribed sphere volume for clothes stacking.

[0066] Understandably, when the clothes are stacked loosely and overlap less, the minimum circumscribed sphere will be relatively large because the space occupied by the clothes is more dispersed. When the clothes overlap closely, the space occupied by the whole will be more compact in three-dimensional space, and the volume of the minimum circumscribed sphere will become smaller.

[0067] It should be added that point cloud data is composed of a large number of discrete points representing the coordinates of each point on the surface of clothing, usually containing x, y, and z three-dimensional coordinate information, and common mathematical algorithms include methods based on geometric optimization, such as iteratively adjusting the center position and radius of the sphere so that the sphere can just contain all the point cloud data points and the radius is as small as possible. The final sphere is the minimum circumscribed sphere. In actual calculations, there are many ready-made point cloud processing libraries such as PCL that can directly call related functions to realize the calculation of the minimum circumscribed sphere. These libraries integrate efficient and accurate algorithms, which can quickly obtain the center coordinates and radius values ​​of the minimum circumscribed sphere, and calculate the minimum circumscribed sphere volume through the sphere volume calculation formula.

[0068] It should be added that the three-dimensional reconstruction technology, such as the point cloud-based reconstruction method, converts the pixels in the depth image into point cloud data in the three-dimensional space, and then constructs the surface shape of the clothing through algorithms such as surface fitting, and then obtains the three-dimensional contour. The above-mentioned registration and fusion is a relatively mature image processing method, and its specific processing process will not be repeated here.

[0069] It should be added that for ordinary household daily clothing, if you want to keep it relatively loose and easy to scan and distinguish each piece of clothing, the reference stacking height can be set at around 20-30 cm, and the specific value can be 30 cm. When the stacked clothes are mainly small pieces, this height can be appropriately lowered to 10-15 cm, and the specific value can be 15 cm, because small pieces of clothing tend to be stacked tightly and seriously block each other, and a lower reference height helps to take measures such as stratification in advance to ensure complete information acquisition. On the contrary, if most of the clothes are heavy coats, blankets and other large items, due to their large size and space occupation, the reference stacking height can be appropriately increased to 50-60 cm, and the specific value can be 60 cm.

[0070] In a specific embodiment, The value can be 2 mm, which is a reference setting for medium-thickness materials, such as thin sweaters to thick wool coats. The thickness varies greatly. The thickness of thin woolen clothes is about 0.5-1 mm, and the thickness of thick ones can reach 3-5 mm or even thicker. That is, the reference thickness can be set at 2 mm. For example, The value can be 0.8.

[0071] Furthermore, in step S5, determining the number of layers includes: extracting the number of layers of clothing from the three-dimensional contour of the overall scanned visual image, recorded as .

[0072] The number of material types is recorded as , statistical stratification number , , To set the number of reference material types, The symbol for rounding up.

[0073] Understandably, ,

[0074] Indicates the compensation factor of the number of material types corresponding to the number of layers of clothing stacked. It indicates the overall posture stratification requirement corresponding to the compensation factor of the number of layers of clothing stacked. The more types of materials there are, the higher the need for separate testing.

[0075] It should be added that the specific execution process of extracting the number of layers of stacked clothes from the three-dimensional contour of the overall scanned visual image is as follows: first, the three-dimensional contour data of the acquired overall scanned visual image is analyzed and processed, and the spatial relationship between different clothing contours is identified, such as the nesting, overlapping and hierarchical distribution of contours in the vertical direction, so as to distinguish the boundaries of each layer of clothes and then count the number of layers of stacked clothes.

[0076] Understandably, in three-dimensional space, the contours of clothes in different layers will show obvious nesting or covering features. The contours of the outer layer of clothes will surround part or all of the contours of the inner layer of clothes, just like nesting dolls, one layer wrapped in another. By detecting the inclusion relationship of this contour, it is possible to determine which clothes are in the outer layer and which are in the inner layer, so as to sort out the order of each layer of clothes and then count the number of layers. For example, if a larger coat contour completely wraps the contours of several shirts and pants, then the coat is the outermost layer, and the shirts and pants are the inner layer. By continuously analyzing this nesting situation, the distribution of each layer of clothes can be gradually determined. At the same time, when clothes are stacked, each layer has a certain height distribution difference in the vertical direction. With the help of the position information of each piece of clothing on the vertical coordinate recorded in the three-dimensional contour, the changes in the contours of clothes in different height intervals are observed. When a new set of clothing contours that are interrelated and have a clear boundary with the upper and lower layers appear in a certain vertical interval, it means that a new layer of clothes has appeared. For example, if some clothing contours at the bottom are within a certain vertical height range, and another group of contours appears upwards, and there is an obvious space interval or boundary distinction between the two groups of contours, it can be judged that these are two different layers of clothing, and the total number of layers can be determined accordingly. In addition, each piece of clothing itself has relatively fixed shape characteristics and continuity of contours, even when stacked. The contours of the same layer of clothing often have similar shape styles, and their contour lines are relatively continuous and smooth, without sudden interruptions and no connection, unless there are special circumstances such as damage, but this can be temporarily ignored in normal scanning scenarios. By analyzing these shape and continuity characteristics of the contours, it is assisted to determine which contours belong to the same layer of clothing, so as to more accurately sort out the number of layers of clothing stacking. Then, by comprehensively using the above-mentioned spatial relationship, shape, vertical distribution and other feature information presented based on the three-dimensional contours, the number of layers of clothing stacking can be extracted from the three-dimensional contours of the overall scanned visual image, providing necessary data support for subsequent clothing processing links.

[0077] When collecting clothing information of the dirty clothing placement area, the embodiment of the present invention determines whether the dirty clothing placement area needs layered detection based on the collected overall scanning visual image, overall spectral data and overall scanning depth image, thereby effectively solving the current problem of insufficient attention to the accuracy of clothing recognition, facilitating accurate grasp of various characteristics of clothing, and the subsequent automatic washing mode, water volume and amount of detergent added of the clothing can all be accurately controlled, thereby reducing misjudgment, and providing a strong guarantee for the subsequent washing effect.

[0078] The environmental information collection module is used to start the sensor terminal carried by the washing machine to collect environmental information.

[0079] Specifically, the sensor terminal includes multiple components such as a temperature sensor and a humidity sensor, wherein the environmental information includes a current temperature value and the like.

[0080] The clothing washing confirmation module is used to confirm the washing operation instruction based on the environmental information.

[0081] Specifically, confirming the washing operation instruction includes: E1, extracting the color of each piece of clothing from the overall scanned visual image, and calculating the overall washing feasibility of the clothing in combination with the types of each material.

[0082] E2. If the feasibility of washing the whole clothes is greater than or equal to the set reference value, the whole clothes are washed as the washing type; if the feasibility of washing the whole clothes is less than the set reference value, the classified washing is washed as the washing type.

[0083] E3. If the cleaning type is overall cleaning, the overall dirtiness of the clothes is counted, and the volume of the fused three-dimensional posture contour is used as the volume of the stacked clothes. The appropriate amount of detergent added for the dirtiness of the clothes under the stacked clothes volume is located from the database as the target detergent addition amount.

[0084] E4. If the cleaning type is classified cleaning, the clothes in the dirty clothes placement area are cleaned and classified based on the set classified cleaning rules, and the high-definition camera, multi-spectral sensor and depth sensor carried by the laundry robot are started to scan the clothes in each cleaning category, and the dirtiness of the clothes in each cleaning category is counted in the same way as the dirtiness of the overall clothes. Similarly, the target detergent addition amount in each cleaning category is analyzed in the same way as the analysis method of the target detergent addition amount under overall cleaning.

[0085] E5. Extract the current temperature from the environmental information and confirm the water temperature for washing clothes.

[0086] E6. The cleaning type, the water temperature for washing clothes, and the target amount of detergent added under the corresponding cleaning type are made into a cleaning operation instruction.

[0087] The embodiments of the present invention effectively make up for the current lack of external environment perception by combining environmental information when confirming washing operation instructions, fully integrate environmental information into the laundry control process, and perform adaptive adjustments, thereby improving the flexibility of automated laundry control while also ensuring the stability of washing quality.

[0088] Furthermore, the statistical feasibility of washing the whole clothes in step E1 includes: E11, comparing the colors of each piece of clothing with the pre-set fading risk index and the color of each piece of clothing corresponding to the dyeing risk index, obtaining the fading risk index and dyeing risk index corresponding to each piece of clothing color, and screening out the maximum fading risk index and maximum staining risk index .

[0089] E12. Extract the permitted mixed clear material category set of each material category from the database. If a material category is in the permitted mixed clear material category set of another material category, treat the material category as a mixed category of the other material category.

[0090] E13. Count the number of mixed types of each material type, select the maximum value, and divide it by the number of material types to obtain the material type cleaning mixing ratio, which is recorded as .

[0091] E14. Calculate the feasibility of overall laundry cleaning , , , and The reference fading risk index, staining risk index and material type cleaning mixing ratio are set respectively.

[0092] In a specific embodiment, in order to facilitate the analysis, the values ​​of each discoloration risk index and each staining risk index are between 0 and 1, that is, and Both can be set to 0.5. The value can be 0.8.

[0093] Furthermore, in step E3, the overall dirtiness of the clothes is calculated, including: based on the overall scanning depth image, determining the stain coverage ratio .

[0094] The depth difference between each stain area and the surrounding normal clothing surface is identified from the overall scan depth image, and the average depth difference is obtained by mean calculation as the target stain thickness, denoted as .

[0095] Count the overall dirtiness of clothes , , and Represent the weights corresponding to the stain coverage ratio and the target stain thickness, respectively. To set the reference stain thickness.

[0096] It should be added that the weights corresponding to the stain coverage ratio and the target stain thickness can be set based on actual experience, experimental verification, or analysis of the importance of these two factors in reflecting the degree of dirtiness. For example, after multiple experimental comparisons, it was found that the stain coverage ratio and the stain thickness are equally important for accurately judging the degree of dirtiness of clothes. For another example, the stain thickness has a greater impact on the degree of dirtiness. and The values ​​are 0.4 and 0.6 respectively. The value can be 0.1 times the thickness of the clothing.

[0097] It should also be added that the depth difference between each stain area and the surrounding normal clothing surface can be identified from the overall scan depth image through image recognition technology. Among them, image recognition technology is a relatively mature technology available, and its specific recognition process will not be repeated here.

[0098] The stain coverage ratio is determined, including: dividing the overall scan depth image into image blocks, finding the area of ​​each stain region in each image block using an edge detection algorithm, and summing the areas to obtain the total area of ​​the stain region.

[0099] The total area of ​​the stain region in each image block is divided by the contour area of ​​the region where each image block is located to obtain the stain ratio of the image block.

[0100] Count the number of image blocks with a stain ratio greater than 0, record it as the number of stain blocks, and divide it by the number of image blocks to record the ratio as .

[0101] The maximum value is selected from the stain ratios of each stain block and recorded as , and the average stain ratio is obtained by mean calculation, denoted as .

[0102] Confirm the stain coverage ratio , , To set the reference stain ratio.

[0103] In a specific embodiment, the area of ​​each stain region in each image block is found using an edge detection algorithm, and specifically, edge detection algorithms such as Canny edge detection and Sobel edge detection can be selected, and The specific value can be 0.6.

[0104] In a specific embodiment, the classification and washing rules usually take into account many factors, such as the material of the clothes, such as cotton, linen, silk, wool, etc., and the color, such as white, light color, dark color, etc. For example, the rules can be set as follows: 1) Material classification: all cotton clothes are classified into one category, silk clothes are classified into another category, and so on, because clothes of different materials have different requirements for water temperature, detergent type, washing method, etc. when washing. For example, silk materials generally require mild detergents and low-temperature gentle washing, while cotton is relatively more tolerant and can adapt to more conventional detergents and slightly higher water temperatures.

[0105] 2) Color classification: Separate white clothes to prevent them from being stained by dark clothes during the washing process. Light-colored clothes such as light pink and light blue are grouped together, and dark-colored clothes such as black, dark blue, and dark red are grouped together.

[0106] It should be added that when there is overlap between color classification and material classification, if the clothes involved contain materials that are easy to fade, such as some dark cotton clothes that are more likely to fade when washed for the first time, or chemical fiber clothes with unstable quality have poor color fastness, in this situation where the risk of staining is high, it is more appropriate to prioritize classification by color. For example, strictly separate all white clothes, regardless of their material, first ensure that white clothes will not be stained by other fading clothes, and then refine the washing process of different white clothes according to the material. For example, white silk clothes can be gently washed with special silk detergents, and white cotton clothes can be washed in the usual cotton way, but be careful to avoid mixing with fading clothes. When there are many types of clothing materials and different materials have very different requirements for washing conditions, it is more reasonable to prioritize material classification. For example, in a pile of clothes that have wool, cotton, silk and other materials with overlapping colors, wool clothes are extremely delicate and have strict requirements on water temperature, detergent type and washing method, such as gentleness and avoiding long-term rubbing. If wool clothes are not separated by material and handled separately, once mixed with other materials by color, they are easily damaged by shrinkage, deformation, etc., even if they appear to be in the same group in terms of color classification. Therefore, material is the primary classification basis at this time, and then the color is subdivided in each material category to make targeted washing arrangements.

[0107] It should be noted that in another specific embodiment, in many practical situations, the classification and washing rules can also be comprehensively considered, without giving absolute priority to one party. For example, first make a preliminary classification roughly by color, separate white, light color, dark color, etc., and then in each color category, quickly pick out special materials with large differences such as wool, silk, etc. according to the material and handle them separately. The remaining ordinary material clothes are further subdivided according to the specific situation. For example, cotton and linen can be classified according to the subsequent degree of dirtiness. In this way, the importance of color classification to prevent staining is taken into account, and the washing risks caused by material differences can be dealt with in a timely manner, and the clothes can be reasonably classified and washed flexibly.

[0108] Furthermore, step E5 confirms the water temperature for washing clothes, including: comparing the degree of dirtiness of the clothes with a set reference degree of dirtiness of the clothes, and if the degree of dirtiness of the clothes is less than or equal to the set reference degree of dirtiness of the clothes, using the washing water temperature initially set for the washing machine as the washing water temperature for washing clothes.

[0109] If the dirtiness of the clothes is greater than the set reference dirtiness of the clothes, the difference between the two is recorded as , and extract the acceptable washing water temperature range of each material type of clothing from the database, and extract the upper limit value of the acceptable washing water temperature range as the bearing water temperature value.

[0110] The lowest value is selected from the load water temperature values ​​of clothes of various materials and used as the reference clothing washing water temperature, which is recorded as ,Will As the water temperature for washing clothes, To set the reference laundry soiling degree.

[0111] In a specific embodiment, The value can be 0.3.

[0112] When confirming the cleaning operation instructions, the embodiment of the present invention realizes further analysis of the cleaning instructions by confirming the cleaning type, the water temperature of the clothes washing and the target detergent addition amount under the corresponding cleaning type, thereby ensuring the flexibility and specificity of the cleaning settings, while also improving the cleaning efficiency and resource utilization.

[0113] The user interaction verification terminal is used to send the cleaning operation instruction to the user for confirmation, and obtain the confirmed cleaning operation instruction.

[0114] It should be added that the laundry robot integrates voice recognition and synthesis technology to realize voice interaction function, allowing users to send instructions to the laundry robot remotely through APP or voice interaction devices, such as smart speakers, such as temporarily pausing laundry, adjusting washing mode, extending drying time, etc., to meet the personalized needs of users in different scenarios. In addition, users can also pre-set their own laundry habits preferences on the APP, such as commonly used washing modes, detergent dosage and drying methods, etc. The system will automatically save and prioritize these personalized settings in subsequent laundry processes, making the laundry service more in line with the user's daily habits.

[0115] The embodiment of the present invention confirms the cleaning operation instruction and sends it to the user for confirmation, thereby making up for the deficiency that the current human-computer interaction relies more on the interactive signals given by the user to realize functions such as reminders and is difficult to fully meet the personalized needs of users, and can fully meet the personalized needs of users.

[0116] The database is used to store the user's current accumulated images of each piece of clothing and the clothing material spectrum data table, and to store the reference clothing thickness of each material type, the acceptable washing water temperature range and the allowed mixed clear material type set.

[0117] The clothes processing execution module is used to control the water inlet valve of the washing machine to open according to the washing operation instruction, and perform corresponding washing, and dry the clothes after the washing is completed.

[0118] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they shall all fall within the protection scope of the present invention.

Claims

1. A household unmanned control and management system based on a laundry robot, characterized in that: It includes a laundry robot body and a laundry control system, and the laundry control system includes: The clothing recognition and analysis module is used to move to the dirty clothing placement area after the laundry robot receives the washing instruction and collect clothing information in the dirty clothing placement area; An environmental information collection module is used to start the sensor terminal carried by the washing machine to collect environmental information; A clothing washing confirmation module is used to confirm washing operation instructions based on environmental information; A user interaction verification terminal is used to send the cleaning operation instruction to the user for confirmation and obtain the confirmed cleaning operation instruction; A database for storing the current accumulated images of each piece of clothing of the user, storing a spectrum data table of clothing materials, and storing a reference clothing thickness, an acceptable washing water temperature range, and a set of permitted mixed clear material types for each material type; The clothes processing execution module is used to control the water inlet valve of the washing machine to open according to the washing operation instruction, and to perform corresponding washing, and to dry the clothes after the washing is completed; The collecting of clothing information in the dirty clothing placement area includes: A high-definition camera placed in the dirty clothes placement area is started to perform an overall scan to obtain an overall scan visual image; The multispectral sensor carried by the laundry robot is activated to perform an overall scan of the dirty clothes placement area to obtain overall spectral data of the scan; Start the depth sensor carried by the laundry robot to perform an overall scan of the dirty clothes placement area to obtain an overall scan depth image; Determine whether the dirty clothes placement area needs layered testing, including: Using image recognition technology to identify the overall scanned visual image, output the three-dimensional outline of the overall scanned visual image, and extract the outline of each piece of clothing from it; For each pair of adjacent clothing contours, the shortest distance between them is calculated and compared with the set reference distance. If the shortest distance between a pair of adjacent clothing is lower than the set reference distance, the pair of adjacent clothing is recorded as a close fitting adjacent clothing pair. The number of close fitting adjacent clothing pairs is counted and divided by the number of adjacent clothing pairs to obtain the close fitting clothing ratio. Based on the outline of each piece of clothing, the area and area of ​​the area enclosed by the outline of each piece of clothing are determined, the overlapping area of ​​each adjacent clothing outline area is calculated, and the overlapping area is divided by the area of ​​the smaller clothing outline in each adjacent clothing outline area to obtain the overlapping ratio of each adjacent clothing outline area, and the maximum value is selected as the target clothing outline area overlapping ratio; Based on the overall scan depth image, confirm the overall posture layering requirements ; Extract the spectral data table of clothing materials from the database, and then extract the spectral feature vector corresponding to each clothing material, extract each spectral feature parameter of each spectral curve from the overall spectral data, and combine them in a certain order to construct a spectral feature vector, and then obtain each material type based on a pre-set material identification algorithm, and count the number of material types; The close-fitting clothing ratio is greater than the set reference close-fitting clothing ratio as condition 1, the target clothing contour area overlap ratio is greater than the set reference clothing contour area overlap ratio as condition 2, the overall posture stratification requirement is greater than the set value as condition 3, and the number of material types is greater than the set reference material type number as condition 4; It is determined whether any of the conditions 1, 2, 3 and 4 are satisfied. If so, the number of layers is determined as a result of the determination, and the mechanical arm of the laundry robot is activated to gently spread the clothes and layer them in sequence according to the number of layers. When each layer of clothing is exposed, it is scanned by the high-definition camera, multi-spectral sensor and depth sensor carried by the laundry robot, and the scanned visual image, spectral data and scanned depth image of each layer are used as the clothing information of the dirty clothing placement area; If it does not exist, then "no" is taken as the judgment result, and the overall scanned visual image, the overall spectral data and the overall scanned depth image are taken as the clothing information of the dirty clothing placement area.

2. The unmanned household control and management system based on a laundry robot as claimed in claim 1, characterized in that: The step of confirming the overall posture layering requirement includes: Based on the overall scanning depth image, the point cloud data of the clothing in three-dimensional space is obtained, and the mathematical algorithm is used to calculate the minimum circumscribed sphere of the clothing point cloud data, and the minimum circumscribed sphere volume is calculated, which is recorded as ; The 3D reconstruction technology is used to construct the overall 3D posture contour of the clothing, and the edge detection algorithm is used to extract the contour of the clothing in the overall scanned depth image, which is then registered and fused with the overall scanned visual image to obtain the fused 3D posture contour. The volume and height of the fused 3D posture contour are extracted and recorded as and ; Extract the reference clothing thickness of each material type from the database, and select the maximum reference thickness, which is recorded as ; Calculate the overall posture layering requirement , , represents a natural constant, To set the minimum external sphere volume of the clothes pile for reference, is the reference stacking height to be set, To set the reference material thickness, To set the reference volume ratio.

3. The unmanned household control and management system based on a laundry robot as claimed in claim 1, characterized in that: The confirmation of the number of layers includes: The number of layers of clothing stacked is extracted from the three-dimensional contour of the overall scanned visual image, denoted as ; The number of material types is recorded as , statistical stratification number , , To set the number of reference material types, The symbol for rounding up.

4. The unmanned household control and management system based on a laundry robot as claimed in claim 1, characterized in that: The confirmation of the cleaning operation instruction includes: Extract the color of each piece of clothing from the overall scanned visual image, and calculate the overall washing feasibility of the clothing based on the types of materials; If the feasibility of washing the whole clothes is greater than or equal to the set reference value, the whole clothes are washed as the washing type; if the feasibility of washing the whole clothes is less than the set reference value, the classified washing is used as the washing type; If the cleaning type is overall cleaning, the overall dirtiness of the clothes is counted, the volume of the fused three-dimensional posture contour is used as the volume of the piled clothes, and the appropriate amount of detergent added for the dirtiness of the clothes under the piled clothes volume is located from the database as the target detergent addition amount; If the cleaning type is classified cleaning, the clothes in the dirty clothes placement area are cleaned and classified based on the set classified cleaning rules, and the high-definition camera, multi-spectral sensor and depth sensor carried by the laundry robot are started to scan the clothes in each cleaning category, and the dirtiness of the clothes in each cleaning category is counted in the same way as the statistical method of the dirtiness of the overall clothes, and the target detergent addition amount in each cleaning category is analyzed in the same way as the analysis method of the target detergent addition amount under overall cleaning; Extracting the current temperature from the environmental information to confirm the water temperature for washing clothes; The cleaning type, the water temperature for washing clothes and the target amount of detergent added under the corresponding cleaning type are made into cleaning operation instructions.

5. The unmanned household control and management system based on a laundry robot as claimed in claim 4, characterized in that: The statistical analysis of the feasibility of overall laundry cleaning includes: Compare the colors of each piece of clothing with the pre-set fading risk index and the color of each piece of clothing corresponding to the dyeing risk index, obtain the fading risk index and dyeing risk index corresponding to each piece of clothing color, and select the maximum fading risk index and maximum staining risk index ; Extracting the permitted mixed clear material category set of each material category from the database, if a material category is in the permitted mixed clear material category set of another material category, treating the material category as a mixed category of the other material category; Count the number of mixed types of each material type, select the maximum value, and divide it by the number of material types to get the material type cleaning mixing ratio, which is recorded as ; Statistical feasibility of overall clothing cleaning , , , and Specifically, the reference fading risk index, staining risk index and material type cleaning mixing ratio are set.

6. The unmanned household control and management system based on a laundry robot as claimed in claim 4, characterized in that: The statistical method for calculating the overall dirtiness of clothes includes: Confirm the stain coverage ratio based on the overall scan depth image ; The depth difference between each stain area and the surrounding normal clothing surface is identified from the overall scan depth image, and the average depth difference is obtained by mean calculation as the target stain thickness, denoted as ; Count the overall dirtiness of clothes , , and Represent the weights corresponding to the stain coverage ratio and the target stain thickness, respectively. To set the reference stain thickness.

7. The unmanned household control and management system based on a laundry robot as claimed in claim 6, characterized in that: The confirmation of the stain coverage ratio comprises: The overall scanned depth image is divided into image blocks, and the area of ​​each stain region in each image block is found by using an edge detection algorithm, and the total area of ​​the stain region is obtained by summing up; The total area of ​​the stain region in each image block is divided by the contour area of ​​the region where each image block is located to obtain the stain ratio of the image block; Count the number of image blocks with a stain ratio greater than 0, record it as the number of stain blocks, and divide it by the number of image blocks to record the ratio as ; The maximum value is selected from the stain ratios of each stain block and recorded as , and the average stain ratio is obtained by mean calculation, denoted as ; Confirm the stain coverage ratio , , To set the reference stain ratio.

8. The unmanned household control and management system based on a laundry robot as claimed in claim 4, characterized in that: The step of confirming the water temperature for washing clothes comprises: The dirtiness of the clothes is compared with the dirtiness of the clothes set as a reference. If the dirtiness of the clothes is less than or equal to the dirtiness of the reference clothes, the washing water temperature initially set by the washing machine is used as the washing water temperature of the clothes; If the dirtiness of the clothes is greater than the set reference dirtiness of the clothes, the difference between the two is recorded as , and extracting the acceptable washing water temperature range of each material type of clothing from the database, and extracting the upper limit value of the acceptable washing water temperature range as the bearing water temperature value; The lowest value is selected from the load water temperature values ​​of clothes of various materials and used as the reference clothing washing water temperature, which is recorded as ,Will As the water temperature for washing clothes, To set the reference laundry soiling degree.

Images