Sweeping robot operation control method and device, sweeping robot and readable medium

Abstract

The invention relates to a sweeping robot operation control method and device, a sweeping robot and a readable medium. The method comprises the steps of obtaining a starting time feature of the sweeping robot, a state feature of a current environment and a cleaning demand feature; on the basis of the starting time feature of the sweeping robot, the state feature of the current environment and the cleaning demand feature, constructing a feature vector through a pre-trained deep reinforcement learning model; and based on the feature vector, performing multi-dimensional feature analysis through the pre-trained deep reinforcement learning model, and obtaining an optimal control strategy of the sweeping robot in the current environment to perform operation control of the sweeping robot. The deep reinforcement learning model is used for feature recognition analysis, the optimal control strategy can be more accurately output, noise output is automatically adjusted according to multi-dimensional factors in the environment, the flexibility is high, the control precision is high, and the noise and the cleaning effect can be better kept in the optimal state in different scenes.

Description

Technical Field

[0001] The present application relates to the field of household appliance control technology, and in particular to a sweeping robot operation control method, device, sweeping robot and readable medium. Background Art

[0002] As smart homes become increasingly popular, sweeping robots, as an important part of household automation, have been favored by consumers for their high efficiency and convenience. However, despite the continuous advancement of technology, sweeping robots on the market still generally face a problem that needs to be solved: the lack of the ability to adaptively adjust noise and cleaning strategies according to the environment. This limitation has largely restricted the further improvement of user experience, making it difficult for sweeping robots to fully meet the diverse needs of users in actual applications.

[0003] Traditional sweeping robots often use preset cleaning modes and noise control strategies. Whether in a quiet bedroom, a noisy living room, or between a spacious living room and a narrow corridor, they often operate in a fixed way. This "one-size-fits-all" working mode may appear too radical or conservative in specific scenarios. It may disturb the rest or work of family members due to excessive noise, and may also leave sanitary dead corners due to insufficient cleaning strength, thus affecting the overall cleaning effect. Although the existing silent mode has alleviated the noise problem to a certain extent, it is often achieved by reducing the motor speed and reducing the suction force. This noise reduction method often sacrifices cleaning efficiency. More importantly, the silent mode is usually unable to be dynamically adjusted according to the actual situation of the environment. For example, when cleaning the bedroom late at night, the user may hope that the sweeping robot can keep as quiet as possible to avoid affecting sleep; during the day or when no one is at home, they may pay more attention to the thoroughness of cleaning and have a relatively high tolerance for noise. However, existing sweeping robots often cannot flexibly switch working modes according to the differences in these specific scenarios, resulting in limited user experience.

[0004] It can be seen that the existing technology still has problems in the control of sweeping robots, such as poor control flexibility and low control accuracy, and is unable to maintain the optimal noise and cleaning effect in different scenarios. Summary of the invention

[0005] The present application provides a sweeping robot operation control method, device, sweeping robot and readable medium to solve the technical problems in the above-mentioned prior art that the sweeping robot control still has poor control flexibility, low control accuracy, and cannot maintain the noise and cleaning effect to the optimal state in different scenarios.

[0006] According to one aspect of an embodiment of the present application, the present application provides a method for controlling the operation of a sweeping robot, the method comprising: obtaining the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning requirement characteristics; based on the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning requirement characteristics, constructing a feature vector through a pre-trained deep reinforcement learning model; based on the feature vector, performing multi-dimensional feature analysis through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot.

[0007] Optionally, the state characteristics of the current environment include noise characteristics, and obtaining the state characteristics of the current environment of the sweeping robot includes: collecting noise data in the current environment of the sweeping robot based on preset noise collection conditions; performing data preprocessing on the noise data to obtain noise characteristic data to be extracted of the current environment; performing noise characteristic extraction on the noise characteristic data to be extracted to obtain the noise characteristics of the current environment of the sweeping robot.

[0008] Optionally, the state characteristics of the current environment include ground material characteristics, and obtaining the state characteristics of the current environment of the sweeping robot also includes: collecting ground data in the current environment of the sweeping robot based on preset material collection conditions; performing data preprocessing on the ground data to obtain data to be extracted of ground material characteristics of the current environment; performing ground material feature extraction on the data to be extracted of ground material characteristics to obtain the ground material characteristics of the current environment of the sweeping robot.

[0009] Optionally, the extracting ground material features from the data to be extracted from the ground material features includes: determining a material feature extraction algorithm based on the data type of the ground data, the data type includes image data and ultrasonic data, and the material feature extraction algorithm includes an image processing algorithm and an ultrasonic detection algorithm; if the data type is the image data, performing feature extraction on the data to be extracted from the ground material features through the image processing algorithm; if the data type is the ultrasonic data, performing feature extraction on the data to be extracted from the ground material features through the ultrasonic detection algorithm; performing feature selection and dimensionality reduction based on the extracted features to obtain the ground material features of the current environment of the sweeping robot.

[0010] Optionally, based on the feature vector, multi-dimensional feature analysis is performed through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot, including: inputting the feature vector into the pre-trained deep reinforcement learning model, calculating the valuation corresponding to each control strategy based on the feature vector, each control strategy including the suction power, brush head speed and operation mode of the sweeping robot; selecting the optimal control strategy based on the valuation corresponding to each control strategy; controlling the suction power, brush head speed and operation mode of the sweeping robot based on the optimal control strategy.

[0011] Optionally, after performing multi-dimensional feature analysis based on the feature vector through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot, the method also includes: predicting the valuation of the current state based on the current state and the control strategy of the sweeping robot based on the main network in the pre-trained deep reinforcement learning model, and predicting the target value estimate of the next state based on the target network in the pre-trained deep reinforcement learning model; determining the loss based on the valuation of the current state and the target value estimate based on a preset loss function; and updating the network weights in the pre-trained deep reinforcement learning model through back propagation based on the loss.

[0012] Optionally, the method also includes: when the sweeping robot is placed in a new environment, obtaining the startup time characteristics, state characteristics and cleaning requirement characteristics of the sweeping robot in the new environment, and detecting whether there is at least one control strategy that meets the characteristics of the new environment; if not, data marking the startup time characteristics, state characteristics and cleaning requirement characteristics of the sweeping robot in the new environment; and retraining the pre-trained deep reinforcement learning model based on the marked data.

[0013] According to another aspect of an embodiment of the present application, the present application provides a sweeping robot operation control device, the device comprising: a data acquisition module, used to acquire the startup time characteristics of the sweeping robot, the state characteristics of the current environment and the cleaning requirement characteristics; a vector construction module, used to construct a feature vector based on the startup time characteristics of the sweeping robot, the state characteristics of the current environment and the cleaning requirement characteristics through a pre-trained deep reinforcement learning model; a strategy control module, used to perform multi-dimensional feature analysis based on the feature vector through the pre-trained deep reinforcement learning model, to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot.

[0014] According to another aspect of an embodiment of the present application, the present application provides a sweeping robot, which implements operation control of the sweeping robot according to the sweeping robot operation control method described above.

[0015] According to another aspect of an embodiment of the present application, the present application provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the steps of the cleaning robot operation control method.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application provides an operation control method for a sweeping robot. By real-time monitoring of environmental noise and ground conditions, the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and multiple dimensions of cleaning demand characteristics are obtained to construct a feature vector, which can improve the analysis and recognition accuracy of the data parameters of the pre-trained deep reinforcement learning model, thereby selecting the optimal control strategy; and by using the deep reinforcement learning model for feature recognition and analysis, the sweeping robot can effectively respond to complex needs in different scenarios, accurately realize intelligent dynamic noise control, and can automatically adjust the noise output according to the environmental noise level while maintaining the cleaning effect. It is not only highly flexible, but also can show the best noise control and cleaning effects in different home environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A schematic diagram of a hardware environment of an optional sweeping robot operation control method provided according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of an optional sweeping robot operation control method according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of another optional sweeping robot operation control method provided according to an embodiment of the present application;

[0023] Figure 4This is a flow chart of an optional step S202 provided according to an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of an optional sweeping robot operation control device provided according to an embodiment of the present application;

[0025] Figure 6 A schematic diagram of an optional computer device structure provided for an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a sweeping robot operation control method is provided.

[0028] like Figure 1 As shown, the sweeping robot operation control method is applicable to Figure 1 In the hardware system environment shown, the system architecture of the hardware system environment includes a terminal device 101, a server 103 and a sweeping robot 105. The terminal device 101 includes a client, and the client includes a smart home system APP.

[0029] The smart home system can be controlled on the terminal device 101, and the sweeping robot can be controlled by using the server and the network. The server 103 is connected to the terminal device 101 and the sweeping robot 105 through the network, and can be used to provide services for the terminal or the client installed on the terminal and the sweeping robot 105, including processing data requests and instructions from the terminal device 101 and the sweeping robot 105 through the server 103 to achieve communication and data exchange between the two. A database 107 can be set on the server or independently of the server to provide data storage services for the server 103. Among them, the network can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0030] The terminal device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc. The server 103 may be a server providing various services, for example, a background server providing support for the web pages displayed on the terminal device 101.

[0031] Among them, the smart home system integrates facilities related to home life by using integrated wiring technology, network communication technology, security technology, automatic control technology, and audio and video technology to build an efficient management system for residential facilities and family schedule affairs. Through the smart home system, users can remotely control the start, pause, and shut down of the sweeping robot. In addition, the system can also automatically adjust the cleaning mode, speed, suction and other controls according to the user's needs and the state parameters of the current environment of the sweeping robot to achieve a more intelligent control process. The smart home system can monitor the operating status of the sweeping robot in real time, including the specific location, working hours, current power, energy consumption data, etc. This information can be fed back to the user in real time through the terminal device 101, so that the user can keep track of the use of the sweeping robot at any time. In addition, the sweeping robot can include a display interface that can display information such as the specific location, working hours, current power, energy consumption data, etc. of the sweeping robot.

[0032] It should be noted that the sweeping robot operation control method provided in the embodiment of the present application is generally executed by a server, a terminal device and / or a sweeping robot, and accordingly, the sweeping robot operation control device is generally arranged in a server, a terminal device or a sweeping robot. Figure 1 The number of the cleaning robots, terminal devices, networks and servers in the embodiment is only for illustration. Any number of cleaning robots, terminal devices, networks and servers may be provided as required.

[0033] like Figure 2 As shown, Figure 2 This is a flow chart of a sweeping robot operation control method provided by an embodiment of the present invention. Taking the sweeping robot operation control method executed by a server as an example, the sweeping robot operation control method includes the following steps:

[0034] Step S202, obtaining the start-up time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics.

[0035] In this embodiment, the sweeping robot can be used in various scenarios, including but not limited to home environments, libraries, shopping malls, office buildings, etc. After the sweeping robot starts working, data collection can begin. The above-mentioned startup time feature can be collected through the time management module integrated in the sweeping robot to obtain the startup time of the sweeping robot. Based on the startup time, the model learning can obtain the working time of the sweeping robot and determine whether it is night or day.

[0036] In this embodiment, the above-mentioned state characteristics may refer to the environmental factors that affect the operation mode of the sweeping robot in the current environment, which are obtained after feature extraction, wherein the environmental factors include but are not limited to the ground material, environmental noise, ground stain degree, spatial layout, obstacles, season and climate, etc. For example, for hard ground tiles, wooden floors, etc., the sweeping robot can usually play a good cleaning effect; for soft ground such as carpets, the cleaning ability of the sweeping robot may be affected to a certain extent, because the fiber structure of the carpet is easy to hide dirt, and the suction force of the sweeping robot on the carpet may not be strong enough to suck out deep dirt. For another example, in winter, due to the cold and dry climate, dust and hair on the ground are easy to accumulate, and the cleaning force of hair and dust is required to be greater, and it is more necessary to enable the anti-static function; in summer, due to high temperature and humidity, stains and oil stains on the ground are easy to form, and it is more necessary to enable the mopping function and the antibacterial and anti-mildew functions. The state characteristics of the current environment can be collected by sensors of corresponding types.

[0037] In this embodiment, the above-mentioned cleaning demand characteristics can be user input, and the cleaning demand may include cleaning mode, cleaning time, selection of cleaning area, noise control, power management, etc. The cleaning demand characteristics can be extracted by identifying and converting the requirements input by the user. Among them, the user can link the terminal device such as a mobile phone and the sweeping robot with the smart home system, and directly control the smart home system through the mobile phone terminal device to realize the remote control of the sweeping robot, which is more convenient and intelligent, and can improve the user experience.

[0038] Step S204: constructing a feature vector through a pre-trained deep reinforcement learning model based on the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics.

[0039] In this embodiment, deep reinforcement learning (DRL) is an artificial intelligence technology that combines deep learning and reinforcement learning, which can combine the representation ability of deep learning with the decision-making ability of reinforcement learning to achieve higher performance. In deep reinforcement learning, a deep neural network is used to learn state representation, while reinforcement learning methods are used to learn strategies. Deep reinforcement learning is an end-to-end perception and control system, and its learning process can be described as follows: at each moment, the agent interacts with the environment to obtain a high-dimensional observation, and uses deep learning methods to perceive the observation to obtain a specific state feature representation. The value function of each action is evaluated based on the expected return, and the current state is mapped to the corresponding action through a certain strategy. The environment responds to this action and obtains the next observation. By continuously looping the above process, the optimal strategy to achieve the goal can be finally obtained.

[0040] Furthermore, the core algorithms of deep reinforcement learning mainly include Deep Q-Network (DQN), Double DQN, Dueling DQN, Policy Gradient, etc. The deep Q network is a model that combines deep learning and Q learning. It uses a deep neural network to learn Q-values ​​to help agents make better decisions. In DQN, two neural networks with the same structure but different parameters are usually used. A main network with the latest parameters is used to predict Q value estimates, and a target network with non-latest parameters is used to predict Q value realities. This setting helps to disrupt correlations and improve the stability of learning. In this embodiment, the pre-trained deep reinforcement learning model is constructed using a DQN network.

[0041] It should be noted that the above-mentioned intelligent agent in this embodiment can refer to the operation control system in the sweeping robot; each action represents the control strategy output by the model; the value function that evaluates each action based on the expected return can select the optimal control strategy; mapping the current state to the corresponding action through a certain strategy, and the environment responding to this action can refer to the cleaning effect achieved by obtaining the environment based on the optimal control strategy after executing the optimal control strategy.

[0042] Furthermore, after obtaining the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics, the intelligent agent in the model can construct a feature vector based on the startup time characteristics, the state characteristics of the current environment, and the cleaning demand characteristics. The feature vector includes the working time of the sweeping robot, multiple state parameters in the current environment, and user needs, which is conducive to the model to perform multi-dimensional fusion analysis, thereby selecting a more accurate operation control strategy and realizing precise operation control of the sweeping robot.

[0043] In some examples, combined Figure 3 As shown, the initial training of the pre-trained deep reinforcement learning model can be completed through an offline data set, including multi-dimensional data such as different noise levels, ground types, and cleaning effects. The conditions for the end of model training may include: Model convergence: When the Q-value function of the deep Q network tends to be stable after a certain number of iterations, that is, the model error of each round of training no longer decreases significantly. It can also include: Performance meets requirements: When running in a test environment, the sweeping robot can effectively balance noise and cleaning effects in a complex environment, and the performance meets the design requirements. For example, the noise is controlled within the set range and the cleaning rate reaches more than 90%. In addition, it can also include: Reward value maximization: In a simulated environment, the cumulative reward value reaches the set target threshold, indicating that the model can continue to take optimal actions.

[0044] Step S206: Based on the feature vector, multi-dimensional feature analysis is performed through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot.

[0045] In this embodiment, based on the constructed feature vector, the feature learning of each dimensional data can be performed through the deep Q value network, and a Q value vector is output according to the learning situation, which represents the expected value of the long-term return that can be obtained by taking a certain action under a specific state. Each element in the Q value vector can represent a possible control strategy, and the optimal control strategy can be selected according to the Q value estimation of the control strategy. Among them, the principle of the DQN network to select the optimal control strategy in different environments can be the optimal balance principle between low noise and high cleaning effect, that is, the purpose of selecting the optimal control strategy based on the current environmental noise, ground material, user needs, etc. is to achieve the best balance between low noise and high cleaning effect. This dynamic adaptive ability is better than the fixed prediction mode of ordinary neural networks.

[0046] In some examples, based on the Q-value estimates corresponding to each control strategy in the output Q-value vector, for the discretized action space, the agent can select the action with the largest Q-value estimate as the optimal control strategy. The discretized action space means that the number of actions that the agent can perform is limited, each action is selected from a predefined, discrete set, and only one action can be selected from the set to execute. For example, the operation control mode of the sweeping robot is discretized into 8 categories, and the action space of the agent is a discrete set containing these 8 categories. When making decisions, the agent will select an action with the largest Q value (the operation control strategy of the sweeping robot) from the Q-value vector as the optimal control strategy to execute.

[0047] In other examples, based on the output Q-value vector and the Q-value estimates of each control strategy, for the continuous action space, the agent can use a greedy strategy or an approximate optimal strategy to select the optimal control strategy. Among them, the continuous action space means that the agent can perform an infinite number of actions, and the action can be any value in the real number range. In this case, the agent cannot simply select an action from a discrete set, but needs to make a decision within the entire real number range. For example, the control strategy of the sweeping robot is continuously adjustable from 1 to 20, and the action space of the agent corresponds to a continuous real number range. When making decisions, the agent can use a greedy strategy or an approximate optimal strategy to select the optimal control strategy, where the greedy strategy means that the agent chooses the action that can maximize the Q value in the current state.

[0048] In other examples, the intelligent body will execute the optimal control strategy to realize the operation control of the sweeping robot, and at the same time observe the execution results, such as washing effect, energy consumption, etc. These execution results can be used as feedback for subsequent training and optimization of the DQN network.

[0049] In some examples, the noise control mode can be divided into multiple levels, for example, divided into level 1, level 2, and level 3 with gradually increasing noise. Quiet periods and noise control modes during quiet periods can be preset. When the model performs feature analysis, it can prioritize the analysis based on the startup time characteristics of the sweeping robot and prioritize the working time period of the sweeping robot. If it is during the preset quiet period, the low noise mode can be enabled first to avoid noise impact on the user, with low noise control as the main focus and cleaning ability as the secondary focus.

[0050] In this embodiment, by real-time monitoring of environmental noise and ground conditions, the start-up time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics of multiple dimensions are obtained to construct feature vectors, which can improve the analysis and recognition accuracy of the data parameters of the pre-trained deep reinforcement learning model, thereby selecting the optimal control strategy. And by using the deep reinforcement learning model for feature recognition and analysis, the sweeping robot can effectively respond to complex needs in different scenarios, accurately realize intelligent dynamic noise control, and can automatically adjust the noise output according to the environmental noise level while maintaining the cleaning effect. It is not only highly flexible, but also can show the best noise control and cleaning effect in different home environments.

[0051] In some optional embodiments, the state characteristics of the current environment include noise characteristics, combined with Figure 4 As shown, in the above step S202: obtaining the state characteristics of the current environment of the sweeping robot includes:

[0052] S2021, collecting noise data in the current environment of the sweeping robot based on preset noise collection conditions;

[0053] S2022, performing data preprocessing on the noise data to obtain noise feature data to be extracted in the current environment;

[0054] S2023: extract noise features from the noise feature data to be extracted to obtain the noise features of the current environment of the sweeping robot.

[0055] In this embodiment, the preset noise collection conditions may include sampling frequency, sampling quantity, sampling location, etc., to ensure that the collected noise data has sufficient resolution and representativeness. For example, the noise data is collected once every 5 seconds, for a total of 5 times, and the image moves 5 meters in different directions each time the sampling occurs. The above noise data may represent the background noise level in the current environment, in decibels (dB). For example, if the current environment of the sweeping robot is detected as a library, the noise collected is 45dB; if the current environment of the sweeping robot is detected as a home kitchen, the noise collected is 70dB.

[0056] In some examples, before noise sampling, it is necessary to select and deploy a noise sensor or a noise detection module. Among them, a noise sensor with high precision and stability can be selected, for example, a noise sensor based on micro-electromechanical system (MEMS) technology, to ensure that the noise level in the environment can be accurately measured. The selected noise sensor can be installed on the sweeping robot to ensure that the detection range of the noise sensor can fully cover the working area of ​​the sweeping robot and accurately reflect the noise level in the environment. The noise sensor can be connected to the data acquisition module in the operation control system of the sweeping robot to ensure that the noise data can be collected in real time and accurately.

[0057] In this embodiment, the above-mentioned data preprocessing of noise data includes: data cleaning: removing outliers, missing values, etc. in the collected noise data to ensure the integrity and accuracy of the data. For outliers, interpolation, mean filling and other methods can be used for processing; for missing values, they need to be filled or deleted according to the specific situation. Data format conversion: converting the collected noise data into a unified format, such as decibels (dB), etc., for subsequent analysis and processing. Data standardization: converting noise data of different dimensions to the same dimension, eliminating differences between different data sources, and improving data comparability and analysis efficiency.

[0058] Furthermore, based on the noise feature data to be extracted of the current environment obtained after preprocessing, noise features can be extracted. Noise extraction may include time domain feature extraction and frequency domain feature extraction, and may also include other feature extractions. For example, according to actual needs, the duration and change trend of noise may also be extracted. Among them, time domain feature extraction includes analyzing the time domain characteristics of the noise feature data to be extracted of the current environment, and extracting statistical features, for example, extracting statistical features such as mean, variance, peak value, and peak-to-peak value. These features can reflect the overall level and fluctuation of noise data. Frequency domain feature extraction includes performing frequency domain analysis such as Fourier transform on the noise feature data to be extracted of the current environment, and extracting the amplitude, phase and other features of each frequency component. These features can reflect the spectral characteristics and frequency distribution of noise data. Finally, the noise features of the current environment of the sweeping robot are obtained.

[0059] In this embodiment, after the noise data of the environment where the sweeping robot is located is collected for feature extraction, it is input into the model built based on the DQN network for training and learning. The model can identify the cleaning needs of different areas according to the noise characteristics of the environment. For example, in areas with louder noise, there may be more dust or debris. At this time, the sweeping robot can automatically increase the suction and brush head speed to improve the cleaning effect. In relatively quiet areas, the suction and brush head speed can be reduced to save energy and reduce noise interference. In addition, the sweeping robot learns through the model built by the DQN network, and can gradually adapt to the noise characteristics of different home environments, which is conducive to the sweeping robot to more intelligently identify different areas of the home, and adjust the cleaning mode according to the characteristics of each area, so as to achieve the effect of dynamically optimizing noise control and provide users with a better cleaning experience. For example, in the bedroom, the sweeping robot may choose a quieter cleaning mode to avoid disturbing the user's rest.

[0060] In some optional embodiments, the state characteristics of the current environment include ground material characteristics. In the above step S202: obtaining the state characteristics of the current environment of the sweeping robot includes:

[0061] S2024, collecting ground data in the current environment of the sweeping robot based on preset material collection conditions;

[0062] S2025, performing data preprocessing on the ground data to obtain ground material feature data to be extracted in the current environment;

[0063] S2026: Extract ground material features from the ground material feature to be extracted data to obtain the ground material features of the current environment of the sweeping robot.

[0064] In some examples, ground data can be collected by sensors, including ultrasonic sensors, infrared sensors, cameras, etc. When collecting data through cameras, the sweeping robot can preset material collection conditions before collecting ground data through the camera, and at least include preset image resolution, number of sampling points, grayscale, color mode, frame rate, exposure time and white balance, sampling direction and other sampling parameters. When collecting data through sensors, for example, ultrasonic sensors, the sweeping robot can preset material collection conditions before collecting ground data through ultrasonic sensors, and at least include ultrasonic frequency, transmission power, detection range, sampling rate, sampling direction, etc. Regardless of the sensor selected, its preset material collection conditions can be preset accordingly based on the parameters of the sensor used.

[0065] Among them, the ultrasonic sensor detects the ground material by sending and receiving ultrasonic waves. The principle is that different materials have different absorption and reflection characteristics of ultrasonic waves. The camera can detect the ground material by collecting images and performing image feature recognition through image recognition algorithms. Among them, the sensor can be installed at the bottom or nearby of the sweeping robot to ensure that the sensor can directly contact the ground and accurately collect data.

[0066] In this embodiment, when the sweeping robot is started, it can move according to a predetermined path or pattern, and the sensor starts to collect ground data of the current environment, and then pre-processes the detected ground data to obtain the ground material feature data to be extracted of the current environment.

[0067] In some examples, when the ultrasonic sensor collects data, the collected ground data will be expressed in the form of ultrasonic data, and the corresponding data preprocessing process may include: Sampling and quantization: by converting the continuous analog ultrasonic signal into a discrete digital signal, the variation range of the discrete digital signal is divided into several intervals, and each interval is replaced by the same integer number, so as to perform digital encoding. Denoising: Ultrasonic signals may be interfered by various noises during the acquisition process, such as Gaussian noise, Poisson noise, etc., which can be denoised by spatial domain filtering and frequency domain filtering. Reduce the impact of noise on the signal-to-noise ratio of the signal, thereby reducing the accuracy of subsequent analysis. Signal enhancement: Signal enhancement technology aims to improve the strength and clarity of ultrasonic signals in order to better identify the characteristics of ground materials. Common signal enhancement methods include grayscale transformation, histogram correction and image sharpening, etc., which expand the dynamic range of the image, enhance the image contrast, and make the edges and details of the image clearer by changing the grayscale value of the pixel point. Data standardization and normalization: In order to facilitate subsequent data analysis and processing, it is usually necessary to standardize and normalize the ultrasonic data, which helps to eliminate the dimensional differences between the data and improve the accuracy of data analysis. Data verification and correction: Verify and correct the data to ensure its accuracy and completeness, including checking the rationality and consistency of the data and whether there are outliers. For outliers, appropriate corrections or elimination are required. For example, a total of 6 sets of ground material hardness data are collected, and the corresponding hardness values ​​are 10, 20, -5, x, 15, and 25. When cleaning the data, the outlier with a hardness of -5 is deleted, and the missing value x is filled with the mean value 17.5 after deleting the outlier.

[0068] In other examples, when the camera is collecting data, the collected ground data will be presented in the form of images or videos, including features such as color, texture, and shape information. The corresponding data preprocessing process includes: Image acquisition and conversion: The image or video data output by the camera is collected and converted into a format suitable for subsequent processing. Image enhancement: Enhance the image, such as improving the brightness, contrast, and sharpness of the image to improve the quality and clarity of the image. Denoising and smoothing: The image may be affected by noise during the acquisition and transmission process, and denoising is required. At the same time, smoothing is required to remove small fluctuations and unevenness in the image. Color correction and equalization: Color correction and equalization are performed to ensure that the color and brightness of the image meet the requirements of subsequent processing.

[0069] Furthermore, after preprocessing all the ground data, the ground material feature extraction can be performed on the ground material feature to be extracted data. When extracting the feature, different feature extraction algorithms can be selected according to the data type to extract the ground material features of the current environment of the sweeping robot. Among them, the ground material usually includes hard ground, including marble, tile, epoxy, corundum, polished cement, etc.; wooden floor, including solid wood floor, composite wood floor, etc.; carpet: including short-pile carpet, low-density pile carpet, fiber carpet, etc. The ground material features include at least hardness, roughness, hygroscopicity, wear resistance, reflectivity, etc.

[0070] In this embodiment, by collecting ground data based on preset material collection conditions, it is beneficial to collect ground data that better meets the set requirements, and it is also convenient to more accurately identify and preprocess the ground data; through data preprocessing, a more standard data set can be obtained, which is convenient for extracting more accurate and representative ground material features.

[0071] In some optional embodiments, the above step S2026 includes:

[0072] Determine a material feature extraction algorithm based on the data type of the ground data, the data type includes image data and ultrasonic data, and the material feature extraction algorithm includes an image processing algorithm and an ultrasonic detection algorithm;

[0073] If the data type is the image data, extracting features of the ground material features to be extracted using the image processing algorithm;

[0074] If the data type is the ultrasonic data, extracting features of the ground material feature data to be extracted by using the ultrasonic detection algorithm;

[0075] Feature selection and dimensionality reduction are performed based on the extracted features to obtain the ground material features of the current environment of the sweeping robot.

[0076] In this embodiment, the sweeping robot can be integrated with a camera and an ultrasonic sensor. The camera can be turned on to collect ground data, the ultrasonic sensor can be used to collect ground data, or the camera and the ultrasonic sensor can be turned on at the same time to collect ground data. When collecting ground data through the camera, the data type collected is image data, which is expressed in the form of an image. When collecting ground data through an ultrasonic sensor, the data type collected is ultrasonic data. Different feature extraction algorithms are used for feature extraction for different data types.

[0077] In some examples, for camera-only acquisition, in order to identify the ground material features, the image of the ground material feature data to be extracted can be segmented to extract the area or object of interest. Then, from the selected area or object, image recognition algorithms such as CNN, image segmentation algorithms, edge detection algorithms, texture analysis techniques, etc. are used to compare and analyze the pixel values, textures, colors and other information in the image to quickly and accurately identify the ground material features, including color, texture, shape, hardness, etc.

[0078] In other examples, for ultrasonic sensor collection only, the echo detection technology in the ultrasonic detection algorithm can be used to extract the echo difference characteristics of the material in the data to be extracted. Among them, the echo refers to the signal received by the detector after the signal is reflected by the object. The echo detection technology uses this principle to extract the characteristics of the ground material by transmitting a signal and receiving the echo signal reflected back, and analyzing the characteristics of the echo signal.

[0079] Furthermore, when extracting features from ultrasonic data, the received echo signal is processed and analyzed. By extracting characteristic parameters of the echo signal, such as amplitude, frequency, phase, etc., the physical properties of the ground material, such as dielectric constant, conductivity, density, etc., can be reflected based on the extracted characteristic parameters. According to the extracted characteristic parameters, the ground material can be identified and classified using machine learning, pattern recognition and other technologies. Since different ground materials have different echo characteristics, different materials can be distinguished by analyzing the echo characteristics. Among them, the difference in echo characteristics is reflected in the difference in amplitude, frequency, phase, etc. The reflection coefficients of different materials are different, resulting in different amplitudes of echo signals. For example, the reflection coefficient of metal materials is higher, and the amplitude of the echo signal is also larger; while the reflection coefficients of materials such as soil and rock are lower, and the amplitude of the echo signal is relatively small. For another example, some materials may have a strong absorption effect on high-frequency signals, resulting in weak high-frequency echo signals; while they have a better reflection effect on low-frequency signals.

[0080] In other examples, when data is collected simultaneously by a camera and an ultrasonic sensor, preprocessing, feature extraction and other operations are performed on the two types of collected ground data at the same time. The following provides a specific example. The sweeping robot is working in a mixed environment containing floor tiles, wooden floors and carpets. The ultrasonic sensor collects echo data of floor tiles, wooden floors and carpets, and the camera collects image data of floor tiles, wooden floors and carpets. The collected ultrasonic sensor data and camera data are preprocessed separately, and the echo difference features of floor tiles, wooden floors and carpets are extracted from the ultrasonic sensor data. The edge features, texture features and other visual features of floor tiles, wooden floors and carpets are extracted from the camera data. Through the feature selection algorithm, the most representative ground material features are selected from each feature for subsequent material recognition. The ground material features are obtained through multiple methods and multiple dimensions and input into the pre-trained deep learning model for feature training, which is conducive to more accurate identification of the ground material in the current environment, and the optimal control strategy is selected according to the material features to accurately control the suction size, brush head speed and cleaning mode of the sweeping robot, so as to ensure low noise while improving the cleaning effect.

[0081] In this embodiment, different types of ground materials will produce different data features. By identifying the collected data type and selecting the corresponding material feature extraction algorithm according to the identified data type to perform corresponding feature extraction, the pertinence and accuracy of the subsequent processing steps can be ensured, and multi-dimensional and accurate data support can be provided for subsequent model training and identification. And by selecting the corresponding material feature extraction algorithm according to the data type, the advantages of various algorithms can be fully utilized to improve the efficiency and accuracy of feature extraction. Through feature selection and dimensionality reduction processing, the most representative ground material features are finally obtained, and the dimension of the data is reduced at the same time, which can not only reduce the calculation amount of subsequent model training, but also improve the generalization ability of the model. After dimensionality reduction, the redundant information in the original data can be effectively removed, so that the model can be more focused on learning the key features of the ground material, which helps the model converge to the optimal solution faster and improves the training efficiency.

[0082] In some optional embodiments, the above step S206 includes:

[0083] S2061, inputting the feature vector into the pre-trained deep reinforcement learning model, and calculating the estimated value corresponding to each control strategy based on the feature vector, each of the control strategies including the suction force, brush head speed and operation mode of the sweeping robot;

[0084] S2062, selecting the optimal control strategy based on the estimated values ​​corresponding to each of the control strategies;

[0085] S2063: Control the suction force, brush head speed and operation mode of the sweeping robot based on the optimal control strategy.

[0086] In this embodiment, the pre-trained deep reinforcement learning model uses a DQN network as a policy network and a value network, including an input layer, a hidden layer, and an output layer, wherein the hidden layer includes at least one layer. The strategy can be updated and the value evaluated by continuously receiving the execution effects of the acquired data and feedback. The input layer receives the feature vector of the state space, the hidden layer is used to extract the deep features of the state space, and the output layer outputs the Q value estimate of each action.

[0087] In this embodiment, when the feature vector is input into the DQN network, it is received through the input layer, wherein the number of neurons in the input layer matches the dimension of the feature vector. For example, the feature vector contains 10 features, and the input layer corresponds to 10 neurons. Then, the feature vector will pass through the forward propagation process of the network, through the nonlinear transformation of the hidden layer, and finally reach the output layer, and the output layer will output a vector, which includes the Q value estimate of each possible action (the valuation corresponding to each control strategy). For each control strategy, its purpose is low noise and high cleaning effect, which can be specifically achieved by controlling the suction force, brush head speed and operation mode performance. Based on the expected return of each control strategy, the value function of each action is evaluated, and the current state is mapped to the corresponding action through a certain strategy. The environment responds to this action and obtains the next observation. By continuously looping the above process, the optimal control strategy for achieving the goal can be finally obtained, and then the suction force, brush head speed and operation mode of the sweeping robot are controlled based on the optimal control strategy.

[0088] The following is a specific example for illustration. The model input is the environment state (State, S) and action (Action, A). The environment state describes the operating environment of the sweeping robot at the current moment, represented by a feature vector, including the environmental noise level: collected by the environmental noise sensor, indicating the background noise level in the current environment, in decibels (dB). Floor material information: detected by sensors, such as hard floors, tiles, carpets, etc. The state encoding corresponding to different materials: hard floors are 1, carpets are 2, tiles are 3, and other unknown floors are 0. Current time period: indicates the working time of the robot, which can be recorded by the built-in clock and used to identify the time period. When only distinguishing between day and night, the corresponding encoding format is: daytime is 1, nighttime is 2. Of course, it can also be customized to be divided into multiple time periods for encoding. Cleaning task requirements: indicates the current cleaning priority, which can be calculated by user settings or historical records. The corresponding encoding format is: normal cleaning is 1, deep cleaning is 2, and fast cleaning is 3. The current robot operation status: includes parameters such as the current suction level, brush head speed (RPM value), and operation mode. The encoding format is low suction level 1, medium 2, and high 3. Based on the above input parameters, a feature vector (state vector) can be constructed. If the current ambient noise is 45dB, the floor is carpet, the time period is night, the task requirement is normal cleaning, the suction level is 2, and the brush head speed is 1500RPM, then the state vector can be expressed as: S = [45, 2, 2, 1, 2, 1500].

[0089] Furthermore, the above actions can refer to a set of operations that the sweeping robot can perform. The action space defines all the operations that the robot may take in a specific state, including: adjusting the suction level, the corresponding encoding method is: low suction level is 1, medium suction level is 2, and high suction level is 3; adjusting the brush head speed, the corresponding encoding method is: reduce to -1, keep to 0, increase to +1; switch cleaning mode, the corresponding encoding method is: silent mode is 1, normal mode is 2, deep cleaning mode is 3. If the sweeping robot selects medium suction, maintains the brush head speed, and enables silent mode, the action vector can be expressed as: A = [2,0,1]. The model outputs the corresponding Q value according to the above input environment state (State, S) and action (Action, A), which represents the expected value of the long-term reward that can be obtained by taking a certain action in a specific state. Each action corresponds to a Q value, and DQN will select the action with the largest Q value in the current state to execute.

[0090] Among them, the action space can be dynamically adjusted according to the current state: Action adjustment rules: When the noise level is high, the action space tends to include options to reduce suction and brush head speed. In night mode, deep cleaning actions are limited. By dynamically adjusting the action space, the robot can avoid invalid or conflicting action combinations. In addition, DQN not only pays attention to the current noise and cleaning effect, but also considers long-term rewards. For example, in order to avoid disturbing noise at night, low-noise actions may be preferred, even if the cleaning efficiency is slightly reduced in the short term.

[0091] In this embodiment, the DQN network has strong learning ability and adaptability. It can learn and adjust according to the parameters in the feature vector to determine the optimal control strategy. It can accurately and dynamically adjust the suction power, brush head speed and operation mode of the sweeping robot, achieve low noise control, and also meet the requirements of using the best cleaning mode on different ground types, thereby improving cleaning efficiency and user experience.

[0092] In some optional embodiments, the method further includes:

[0093] Based on the main network in the pre-trained deep reinforcement learning model, predicting the estimated value of the current state according to the current state and the control strategy of the sweeping robot, and predicting the target value estimate of the next state based on the target network in the pre-trained deep reinforcement learning model;

[0094] Determining a loss based on a preset loss function according to an estimate of the current state and the target value;

[0095] Based on the loss, the network weights in the pre-trained deep reinforcement learning model are updated through back propagation.

[0096] In some examples, combined Figure 3 As shown in the figure, in the DQN network, an experience replay buffer can be used to store the experience of the agent's interaction with the environment. At the same time, a target network with the same structure as the main network but fixed parameters is introduced. The parameters of the target network are regularly updated to the parameters of the main network to reduce the fluctuation of the target value caused by parameter updates. In each training round, the agent selects an action based on the current state and strategy. After executing the action, observe the environmental feedback (i.e., the next state and reward). The current state, action, reward, and next state are stored in the experience replay buffer, and a batch of experience is randomly extracted from the experience replay buffer for training the DQN network. The Q value is calculated using the target network, and the parameters of the DQN network are optimized using the back-propagation algorithm. Repeat the above process from initialization to back-propagation optimization of network parameters until the DQN network converges or reaches the preset training round.

[0097] During the training process, the training goal of the DQN network is to make the Q value (target value estimate) output by the network closer to the actual Q value (current estimate). In order to reduce the error between the two, DQN can use a loss function to measure the error between the Q value output by the network and the target Q value. Among them, the loss function can include mean square error (MSE) or other error measurement methods.

[0098] In some examples, the above DQN network can use two neural networks: one is the main network (Eval Net), which is used to select actions and predict Q values; the other is the target network (Target Net), which is used to calculate the target Q value. Among them, the action is also the control strategy of the sweeping robot, and the target Q value is calculated as follows: Use the main network to predict the Q value of the current state S(t), and select an action A according to the ε-greedy strategy. Execute this action and observe the next state S(t+1) and the reward R obtained. Use the target network to calculate the maximum predicted Q value of the next state S(t+1), that is, max Q(S(t+1), a'), where a' is all possible actions in the next state S(t+1). Combined with the reward R of the current state to calculate the target Q value, that is, the target Q value y=R+γ*max Q(S(t+1), a'), where γ is a discount factor, indicating the degree of discount of future rewards. Then, calculate the error (loss) between the Q value Q(S(t),A) predicted by the main network and the target Q value y, that is, L = (Q(S(t),A)-y) 2 .

[0099] Furthermore, after obtaining the loss value, the DQN network uses the back propagation algorithm to update the network weights of the main network to minimize the error L. This process is repeated until the network converges to a stable strategy, thereby achieving model optimization, which is conducive to more accurate feature training and recognition of the model. Among them, the method of updating the network weights includes but is not limited to: calculating the gradient of the loss function with respect to the network output; using the chain rule to calculate the gradient of the loss function with respect to the network weights; using the gradient descent algorithm to update the network weights.

[0100] In this embodiment, by calculating the error of the estimated outputs of the two networks in the DQN network and updating the network weights of the model based on the error back propagation, the weight parameters of the DQN network can be optimized to make it more stable and accurately predict, achieve policy optimization, and help the intelligent agent make more intelligent decisions when selecting actions. In addition, when updating the weights of the main network through back propagation, since the target network remains unchanged, the problem of training instability caused by frequent changes in network weights can be avoided.

[0101] In some optional embodiments, the method further includes:

[0102] When the sweeping robot is placed in a new environment, the start-up time characteristics, state characteristics and cleaning demand characteristics of the sweeping robot in the new environment are obtained, and it is detected whether there is at least one control strategy that meets the characteristics of the new environment;

[0103] If not, data marking is performed on the startup time characteristics of the sweeping robot in the new environment, the state characteristics of the new environment, and the cleaning requirement characteristics;

[0104] The pre-trained deep reinforcement learning model is retrained based on the labeled data.

[0105] In this embodiment, when the sweeping robot is used in some new environments, the model will lack the learning of the environmental data of the new environment. Therefore, when the strategy output is based on the startup time characteristics, the state characteristics of the current environment and the cleaning demand characteristics obtained when the sweeping robot is in the new environment, there may be a situation where the strategy is not optimal or even no control strategy can be applied to the new environment. In this regard, the startup time characteristics of the sweeping robot collected in the new environment, the state characteristics of the new environment and the cleaning demand characteristics can be data labeled, and the labeled data can be constructed as a new training data set, which is used as the optimization data of the pre-trained deep learning model for model optimization to improve the model's adaptability to more scenarios. For example: the sweeping robot has not been used in an extremely low environment before, and after changing to a new environment, it needs to meet the requirements of extremely low noise. Under such requirements, the cleaning force requirements will be lowered during model training to meet the extremely low noise requirements.

[0106] In this embodiment, by detecting whether the control strategy output by the model can meet the requirements of various features in the new environment, data marking is performed if it is not satisfied and the marked data is used as a new training data set to optimize the optimization data of the pre-trained deep learning model. This is beneficial to improving the model's adaptability to more scenarios, thereby ensuring that the sweeping robot provides better noise control and cleaning capabilities in more scenarios.

[0107] According to another aspect of the embodiment of the present application, Figure 5 As shown, corresponding to the sweeping robot operation control method in the above embodiment, this embodiment provides a sweeping robot operation control device, the device comprising:

[0108] The data acquisition module 501 is used to acquire the start-up time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics;

[0109] A vector construction module 503 is used to construct a feature vector through a pre-trained deep reinforcement learning model based on the startup time characteristics of the sweeping robot, the state characteristics of the current environment and the cleaning demand characteristics;

[0110] The strategy control module 505 is used to perform multi-dimensional feature analysis based on the feature vector through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot.

[0111] It should be noted that, in this embodiment, the data acquisition module 501 can be used to execute step S202 in the embodiment of the present application, the vector construction module 503 in this embodiment can be used to execute step S204 in the embodiment of the present application, and the policy control module 505 in this embodiment can be used to execute step S206 in the embodiment of the present application.

[0112] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0113] According to another aspect of the embodiments of the present application, the present application provides a sweeping robot, which implements the operation control of the sweeping robot according to the above-mentioned sweeping robot operation control method, and can implement various embodiments of the sweeping robot operation control method and achieve corresponding technical effects, which will not be repeated here.

[0114] Optionally, an operation control system is integrated in the sweeping robot. The sweeping robot may be integrated with a camera, an environmental noise sensor, a ground material detection module, a time management module, etc. The camera, the environmental noise sensor, the ground material detection module, the time management module, etc. interact with the operation control system for data, including data request, motion control, data collection, etc.

[0115] According to another aspect of the embodiment of the present application, the present application provides a computer device, such as Figure 6 As shown, it includes a memory 601, a processor 603, a communication interface 605 and a communication bus 607. The memory 601 stores a computer program that can be run on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 607. When the processor 803 executes the computer program, the steps of the above-mentioned sweeping robot operation control method are implemented.

[0116] The memory and processor in the above-mentioned computer device communicate through a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0117] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the processor.

[0118] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0119] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the sweeping robot operation control method in any of the above embodiments.

[0120] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for the processor to execute the steps of the sweeping robot operation control method described in the above embodiment.

[0121] Optionally, the specific examples in this embodiment can refer to the examples described in the above embodiments, and this embodiment will not be repeated here. And when the embodiments of this application are specifically implemented, they can refer to the above embodiments and have corresponding technical effects.

[0122] It is understood that the embodiments described herein can be implemented with hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof. For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in a processor or outside a processor.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0125] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0126] It should be noted that, in this article, relational terms such as first, second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0127] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A sweeping robot operation control method, characterized in that: The method comprises: Obtain the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics; Based on the startup time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics, a feature vector is constructed through a pre-trained deep reinforcement learning model; Based on the feature vector, multi-dimensional feature analysis is performed through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot.

2. The sweeping robot operation control method according to claim 1, characterized in that: The state characteristics of the current environment include noise characteristics, and obtaining the state characteristics of the current environment of the sweeping robot includes: Collect noise data in the current environment of the sweeping robot based on preset noise collection conditions; Performing data preprocessing on the noise data to obtain noise feature data to be extracted in the current environment; Noise feature extraction is performed on the noise feature to be extracted data to obtain the noise feature of the current environment of the sweeping robot.

3. The sweeping robot operation control method according to claim 1, characterized in that: The state characteristics of the current environment include ground material characteristics, and obtaining the state characteristics of the current environment of the sweeping robot also includes: Collect ground data in the current environment of the sweeping robot based on preset material collection conditions; Preprocessing the ground data to obtain the ground material feature data to be extracted in the current environment; The ground material feature extraction is performed on the ground material feature to be extracted data to obtain the ground material feature of the current environment of the sweeping robot.

4. The sweeping robot operation control method according to claim 3, characterized in that: The extracting of ground material features from the ground material feature to be extracted data comprises: Determine a material feature extraction algorithm based on the data type of the ground data, the data type includes image data and ultrasonic data, and the material feature extraction algorithm includes an image processing algorithm and an ultrasonic detection algorithm; If the data type is the image data, extracting features of the ground material features to be extracted using the image processing algorithm; If the data type is the ultrasonic data, extracting features of the ground material feature data to be extracted by using the ultrasonic detection algorithm; Feature selection and dimensionality reduction are performed based on the extracted features to obtain the ground material features of the current environment of the sweeping robot.

5. The sweeping robot operation control method according to claim 1, characterized in that: The method of performing multi-dimensional feature analysis based on the feature vector through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot includes: Input the feature vector into the pre-trained deep reinforcement learning model, and calculate the valuation corresponding to each control strategy based on the feature vector, each of the control strategies including the suction force, brush head speed and operation mode of the sweeping robot; Selecting the optimal control strategy based on the estimated values ​​corresponding to each of the control strategies; The suction force, brush head speed and operation mode of the sweeping robot are controlled based on the optimal control strategy.

6. The sweeping robot operation control method according to claim 5, characterized in that: After performing multi-dimensional feature analysis based on the feature vector through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot, the method further includes: Based on the main network in the pre-trained deep reinforcement learning model, predicting the estimated value of the current state according to the current state and the control strategy of the sweeping robot, and predicting the target value estimate of the next state based on the target network in the pre-trained deep reinforcement learning model; Determining a loss based on a preset loss function according to an estimate of the current state and the target value; Based on the loss, the network weights in the pre-trained deep reinforcement learning model are updated through back propagation.

7. The sweeping robot operation control method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the sweeping robot is placed in a new environment, the start-up time characteristics, state characteristics and cleaning demand characteristics of the sweeping robot in the new environment are obtained, and it is detected whether there is at least one control strategy that meets the characteristics of the new environment; If not, data marking is performed on the startup time characteristics of the sweeping robot in the new environment, the state characteristics of the new environment, and the cleaning requirement characteristics; The pre-trained deep reinforcement learning model is retrained based on the labeled data.

8. A sweeping robot operation control device, characterized in that: The device comprises: The data acquisition module is used to obtain the start-up time characteristics of the sweeping robot, the state characteristics of the current environment, and the cleaning demand characteristics; A vector construction module, used to construct a feature vector through a pre-trained deep reinforcement learning model based on the startup time characteristics of the sweeping robot, the state characteristics of the current environment and the cleaning demand characteristics; The strategy control module is used to perform multi-dimensional feature analysis based on the feature vector through the pre-trained deep reinforcement learning model to obtain the optimal control strategy of the sweeping robot in the current environment to control the operation of the sweeping robot.

9. A sweeping robot, characterized in that: The sweeping robot implements operation control of the sweeping robot according to the sweeping robot operation control method described in any one of claims 1 to 7.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the steps of the sweeping robot operation control method according to any one of claims 1 to 7.

Images