Control method and device and robot

Abstract

The invention provides a control method and device and a robot, and is applied to the robot, and the method comprises the steps: obtaining multi-modal data corresponding to a target scene; the target scene is a scene where the robot is located in the process of following the target object; based on the multi-modal data, determining a stain state corresponding to the easy-to-slip stain in the target scene; generating risk assessment information of the target object based on the motion trend of the target object and the stain state corresponding to the easy-to-slip stain; and generating a to-be-executed target strategy based on the motion trend and the stain state of the target object under the condition that the risk assessment information represents that the easy-to-slip stain causes the target object to have a safety risk.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of data processing technology, and in particular, to a control method, device, and robot. Background Art

[0002] When there are slippery stains at home, the elderly may slip after coming into contact with the slippery stains, posing a relatively high safety risk. Summary of the Invention

[0003] In view of this, embodiments of this application at least provide a control method, device, robot, and storage medium.

[0004] The technical solution of the embodiments of this application is implemented as follows:

[0005] Embodiments of this application provide a control method applied to a robot. The method includes:

[0006] Obtain multi-modal data corresponding to a target scenario; the target scenario is the scenario where the robot is located during the process of following a target object;

[0007] Based on the multi-modal data, determine the stain state corresponding to the slippery stain in the target scenario;

[0008] Based on the movement trend of the target object and the stain state corresponding to the slippery stain, generate risk assessment information for the target object;

[0009] In the case where the risk assessment information indicates that the slippery stain causes a safety risk to the target object, generate a target policy to be executed based on the movement trend of the target object and the stain state.

[0010] Embodiments of this application provide a control device applied to a robot. The device includes:

[0011] An obtaining unit, configured to obtain multi-modal data corresponding to a target scenario; the target scenario is the scenario where the robot is located during the process of following a target object;

[0012] A determining unit, configured to determine the stain state corresponding to the slippery stain in the target scenario based on the multi-modal data;

[0013] A first generating unit, configured to generate risk assessment information for the target object based on the movement trend of the target object and the stain state corresponding to the slippery stain;

[0014] A second generating unit, configured to generate a target policy to be executed based on the movement trend of the target object and the stain state in the case where the risk assessment information indicates that the slippery stain causes a safety risk to the target object.

[0015] An embodiment of the present application provides a robot, which includes a processor, an image acquisition device, and an actuator, where:

[0016] The image acquisition device is configured to acquire multimodal data corresponding to a target scene; the target scene is the scene where the robot is located during the process of following a target object;

[0017] The processor is configured to determine a stain state corresponding to a slippery stain in the target scene based on the multimodal data; generate risk assessment information of the target object based on the motion trend of the target object and the stain state corresponding to the slippery stain; and generate a target policy to be executed based on the motion trend of the target object and the stain state when the risk assessment information indicates that the slippery stain causes a safety risk to the target object.

[0018] The actuator is configured to execute the target policy.

[0019] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method is implemented.

[0020] In the embodiment of the present application, the stain state corresponding to the slippery stain in the target scene is determined through the acquired multimodal data, and then the risk assessment information of the target object is generated based on the motion trend of the target object and the stain state corresponding to the slippery stain. Thus, when the risk assessment information indicates that the slippery stain causes a safety risk to the target object, the target policy to be executed is generated based on the motion trend of the target object and the stain state. In this way, on the one hand, compared with determining the stain state through single-modal data, since the stain state is determined through multimodal data, the misjudgment probability is reduced while the accuracy of determining the stain state is improved; on the other hand, since the risk assessment information is related to the motion trend of the target object and the stain state, corresponding risk assessment information can be generated according to different actual application scenarios, so that the generated risk assessment information can be more flexible and closer to the current application scenario; on the other hand, since the target policy to be executed is generated through the motion trend and the stain state, the corresponding target policy can be selected in real time according to different actual scenarios, which improves the decision-making ability and intelligent level of the robot. At the same time, by executing the target policy, the safety risk of the target object can be reduced, and the safety of the target object is improved.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the technical solution of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and form a part of this specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to explain the technical solution of the present application.

[0023] Figure 1 Schematic diagram of the implementation process of a control method provided by an embodiment of the present application Figure 1 ;

[0024] Figure 2 Schematic diagram of the implementation process of a control method provided by an embodiment of the present application Figure 2 ;

[0025] Figure 3 Schematic diagram of the composition structure of a control device provided by an embodiment of the present application;

[0026] Figure 4 Schematic diagram of the hardware entity of a robot provided by an embodiment of the present application. Detailed implementation manners

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0028] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0029] It should be noted that the terms "first / second / third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0030] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as the general understanding of those of ordinary skill in the technical field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0031] An embodiment of the present application provides a control method. By acquiring multimodal data, the stain state corresponding to the slippery stain in the target scene is determined. Then, based on the movement trend of the target object and the stain state corresponding to the slippery stain, risk assessment information of the target object is generated. Thus, when the risk assessment information indicates that the slippery stain causes a safety risk to the target object, a target strategy to be executed is generated based on the movement trend of the target object and the stain state. On the one hand, compared with determining the stain state through data of a single modality, since the stain state is determined through multimodal data, the probability of misjudgment is reduced while the accuracy of determining the stain state is improved. On the other hand, since the risk assessment information is related to the movement trend of the target object and the stain state, corresponding risk assessment information can be generated according to different actual application scenarios, making the generated risk assessment information more flexible and closer to the current application scenario. On the other hand, since the target strategy to be executed is generated through the movement trend and the stain state, the corresponding target strategy can be selected in real time according to different actual scenarios, improving the decision-making ability and intelligent level of the robot. At the same time, by executing the target strategy, the safety risk of the target object can be reduced, improving the safety of the target object.

[0032] Figure 1 Schematic implementation process of a control method provided by an embodiment of the present application Figure 1 As Figure 1 shown, this method is applied to a robot and includes steps S101 to S104, where:

[0033] Step S101, acquire multimodal data corresponding to the target scene; the target scene is the scene where the robot is located during the process of following the target object.

[0034] Here, the target scene can be any suitable scene, for example, indoor, outdoor, etc. Multimodal data refers to data from different devices or data with different data formats. Multimodal data can include but is not limited to at least one of: image data, audio data, etc. Image data can include but is not limited to at least one of: infrared images, visual images, etc. The target object can be any suitable object, for example, a person, a robot, etc.

[0035] In some embodiments, the multimodal data corresponding to the target scene can be directly acquired by the robot through a configured image acquisition device and / or sensor. The image acquisition device is a device with the functions of acquiring data, recording data, and transmitting data. The image acquisition device can include but is not limited to at least one of: a camera, a radar system, etc. The sensor can include but is not limited to at least one of: a microphone, a temperature sensor, etc.

[0036] In some embodiments, when the multimodal data includes image data, the method for obtaining the image data may include, but is not limited to: reading the collected data from an image acquisition device to obtain the image data, receiving the collected data sent by the image acquisition device to obtain the image data, and the like. For example, the image acquisition device has the function of storing data, and the robot can obtain the image data by reading the data in the memory of the image acquisition device. Also for example, the image acquisition device has the function of transmitting data, and by establishing a communication connection between the image acquisition device and the robot, the robot can receive the data sent by the image acquisition device to obtain the image data.

[0037] Step S102: Based on the multimodal data, determine the stain state corresponding to the slippery stain in the target scene.

[0038] Here, the slippery stain is a stain that can easily cause the target object to slip. The slippery stain may include, but is not limited to, at least one of water stains, oil stains, and the like.

[0039] In some embodiments, since the slippery stain can be regarded as a flowing state or whether the flow of the slippery stain is not considered, the stain state corresponding to the slippery stain may include, but is not limited to, one of the following: the target stain area where the slippery stain is located, the change trend of the slippery stain, and the like. For example, when the slippery stain is regarded as a flowing state, the stain state corresponding to the slippery stain may include the change trend of the slippery stain. Also for example, when whether the slippery stain flows is not considered, the stain state corresponding to the slippery stain may include the target stain area where the slippery stain is located.

[0040] In some embodiments, a preset processing method may be used to process the multimodal data to obtain the stain state. The preset processing method may be any form including, but not limited to, at least one of the following: algorithms, models, neural networks, and the like. For example, the preset processing method may be a neural network equipped with a processing algorithm, or an image processing model. Any object that can implement the function of processing multimodal data can be used as the preset processing method, and the form of the preset processing method is not limited in this application.

[0041] In some embodiments, since the multimodal data may include at least one type of data, weights can be assigned to different data according to the importance of different data, and then according to the assigned weights, the proportion of different data in the multimodal data can be allocated, so that the stain state can be determined more accurately. In implementation, since the image data can more intuitively reflect the situation of the stain, the image data, as the more important data, can be assigned a larger weight value. In some embodiments, the value of the weight can be within a certain range. For example, the value of the weight can be within [0, 1], and the weight value can be 0.35, 0.6, etc.

[0042] In some embodiments, when the multimodal data includes an infrared image and a visual image, since the visual image is a three-channel image and the infrared image is a single-channel image, the infrared image and the visual image can be fused to obtain a four-channel image, and then the stain state corresponding to the slippery stain in the target scene can be determined according to the four-channel image.

[0043] In some embodiments, when the stain state includes the change trend of the slippery stain, the multimodal data at at least one acquisition moment can be determined, so that the change trend of the slippery stain can be determined based on the multimodal data at at least one acquisition moment.

[0044] In some embodiments, when the stain state includes the target stain area where the slippery stain is located, the multimodal data includes an infrared image and a visual image. The first stain area can be determined based on the infrared image, and the second stain area can be determined based on the visual image, so that the target stain area can be determined based on the first stain area and the second stain area.

[0045] Step S103, generate risk assessment information of the target object based on the motion trend of the target object and the stain state corresponding to the slippery stain.

[0046] Here, the motion trend of the target object is used to characterize the change of the motion parameters of the target object over time. The motion trend of the target object can characterize that the target object is in a stationary state or a motion state. The risk assessment information of the target object can be used to characterize the probability that the target object passes through the location of the slippery stain or the probability that the target object slips. In some embodiments, the risk assessment information can be represented in any form including but not limited to percentages, decimals, etc. For example, the risk assessment information can be 81%, 0.6, etc. In some embodiments, the value of the risk assessment information is within a certain range. For example, the value of the risk assessment information can be within [0, 1].

[0047] In some embodiments, a first correspondence relationship can be established among the motion trend, the stain state, and the risk assessment information, so that after the motion trend and the stain state are determined, the risk assessment information of the target object can be generated according to the first correspondence relationship.

[0048] In some embodiments, the risk assessment model can be trained according to the motion trend, the stain state, and the risk assessment information at historical moments, and then the motion trend and the stain state can be input into the trained risk assessment model, and the risk assessment model can output the risk assessment information.

[0049] In some embodiments, when the motion trend indicates that the target object is in a stationary state, the risk assessment information may indicate that there is no safety risk for the target object caused by the slippery stain. When the motion trend indicates that the target object is in a moving state, the activity area of the target object can be determined, and then the risk assessment information can be generated based on the activity area of the target object and the stain state corresponding to the slippery stain.

[0050] In some embodiments, when the stain state includes the change trend of the slippery stain, the risk assessment information of the target object can be generated based on the motion trend of the target object and the change trend of the slippery stain. Among them, the risk assessment information is used to represent the probability that the target object passes through the slippery stain in the changing process.

[0051] In some embodiments, when the stain state includes the target stain area where the slippery stain is located, the first probability that the target object passes through the target stain area and the second probability that the target object contacts the target stain area can be determined based on the motion trend of the target object and the target stain area, and the risk assessment information of the target object can be generated based on the first probability and the second probability.

[0052] Step S104, when the risk assessment information indicates that there is a safety risk for the target object caused by the slippery stain, a target strategy to be executed is generated based on the motion trend of the target object and the stain state.

[0053] Here, the target strategy to be executed is used to reduce the safety risk of the target object. The target strategy to be executed may include, but is not limited to, one of the first preset strategy, the second preset strategy, the third preset strategy, etc.

[0054] In some embodiments, the first preset strategy may include, but is not limited to, at least one of the following: the closing method corresponding to the target valve, the wiping strategy corresponding to the slippery stain. The second preset strategy is used to remind the target object to adjust the motion mode. The third preset strategy may include, but is not limited to, at least one of the following: locating the water leakage position, analyzing the cause of the water leakage, the processing strategy of the target object.

[0055] In some embodiments, when the risk assessment information meets the probability condition, it can be determined that the risk assessment information indicates that there is a safety risk for the target object caused by the slippery stain. The probability condition may include, but is not limited to: being greater than the preset probability value, not less than the preset probability value, etc. In some embodiments, the preset probability value can be represented in any form including, but not limited to: percentages, decimals, etc. For example, the preset probability value can be 75%, 0.5, etc. In some embodiments, the value of the preset probability value is within a certain range. For example, the value of the preset probability value can be within [0, 1].

[0056] The method for generating the target policy may include but is not limited to: generating the target policy based on the motion trend and stain state of the target object through machine learning, generating the target policy based on the motion trend and stain state of the target object through multi-objective optimization, etc. For example, the policy generation rules can be learned from historical data, and then the rules are used to train the policy generation model. Then, the motion trend and stain state of the target object are input into the policy generation model, and the output of the model is the target policy. Among them, the policy generation model can be any suitable model, such as a supervised learning model, a reinforcement learning model, a deep learning model, etc. Another example is that the policy generation problem can be modeled as a multi-objective optimization problem based on the motion trend and stain state of the target object, and then an optimization algorithm is used to solve the multi-objective optimization problem to generate the target policy. Among them, the objectives may include but are not limited to at least one of maximizing the safety of the target object, minimizing the energy consumption of the robot, minimizing the execution time of the policy, etc. The optimization algorithm may include but is not limited to: genetic algorithm, particle swarm optimization algorithm, etc.

[0057] In some embodiments, when the risk assessment information indicates that there is no safety risk for the target object caused by the slippery stain, the multi-modal data corresponding to the target scene can be obtained again, and the stain state corresponding to the slippery stain in the target scene is determined based on the multi-modal data. Then, based on the motion trend of the target object and the stain state corresponding to the slippery stain, the risk assessment information of the target object is generated again, so that the safety risk of the target object can be continuously monitored.

[0058] In some embodiments, when the motion trend indicates that the target object is in a stationary state, the generated first preset policy can be used as the target policy. When the motion trend indicates that the target object is in a moving state, the activity area of the target object can be determined, and based on the activity area of the target object and the change trend of the slippery stain, the generated second preset policy can be used as the target policy.

[0059] In the embodiments of the present application, based on the acquired multimodal data, the stain state corresponding to the slippery stain in the target scene is determined, and then based on the movement trend of the target object and the stain state corresponding to the slippery stain, risk assessment information of the target object is generated. Thus, when the risk assessment information indicates that the target object has a safety risk caused by the slippery stain, based on the movement trend of the target object and the stain state, a target strategy to be executed is generated. On the one hand, compared with determining the stain state through single-modal data, since the stain state is determined through multimodal data, the probability of misjudgment is reduced while the accuracy of determining the stain state is improved. On the other hand, since the risk assessment information is related to the movement trend and stain state of the target object, corresponding risk assessment information can be generated according to different actual application scenarios, making the generated risk assessment information more flexible and closer to the current application scenario. On the other hand, since the target strategy to be executed is generated through the movement trend and stain state, the corresponding target strategy can be selected in real time according to different actual scenarios, improving the decision-making ability and intelligent level of the robot. At the same time, by executing the target strategy, the safety risk of the target object can be reduced, improving the safety of the target object.

[0060] In some embodiments, the stain state includes the change trend of the slippery stain. When determining the stain state corresponding to the slippery stain in the target scene, the following steps may be performed:

[0061] Step S121, determine the multimodal data at at least one acquisition moment.

[0062] Here, the acquisition moment refers to the moment when the multimodal data is acquired. In some embodiments, the multimodal data corresponding to different acquisition moments may be the same or different.

[0063] In some embodiments, the change trend of the slippery stain can be determined by acquiring the multimodal data at one acquisition moment. However, the determined change trend of the slippery stain may not be accurate enough. Therefore, the change trend of the slippery stain can be determined by acquiring the multimodal data at at least two acquisition moments.

[0064] In some embodiments, the multimodal data at at least one acquisition moment can be obtained by the robot through the configured image acquisition device and / or sensor. During implementation, the data at at least one acquisition moment can be read from the image acquisition device and / or sensor to determine the multimodal data at at least one acquisition moment. The image acquisition device may include, but is not limited to, at least one of a camera, a radar system, etc. The sensor may include, but is not limited to, at least one of a microphone, a temperature sensor, etc.

[0065] Step S122: Determine the change trend of the slippery stain based on the multi-modal data at at least one acquisition moment; the change trend is used to determine the attribute of the slippery stain at a certain acquisition moment.

[0066] Here, the change trend of the slippery stain is used to determine the attribute of the slippery stain at a certain acquisition moment through the multi-modal data at at least one acquisition moment. The state of the slippery stain over a period of time can be determined. The change trend of the slippery stain can include, but is not limited to, one of the following: increase, decrease, unchanged, etc.

[0067] In some embodiments, a preset processing method can be used to process the multi-modal data at at least one acquisition moment to obtain the change trend of the slippery stain. The preset processing method can be any form including but not limited to at least one of the following: algorithms, models, neural networks, etc.

[0068] In some embodiments, when the multi-modal data at at least one acquisition moment includes image data, the image data can be sorted in the order of at least one acquisition moment to determine the change in the presented area of the slippery stain in the image data, and then the change trend of the slippery stain can be determined according to the change in the presented area. For example, when the presented area of the slippery stain at at least one acquisition moment does not change, the change trend of the slippery stain can be unchanged.

[0069] In some embodiments, when the multi-modal data at at least one acquisition moment includes audio data, the audio source of the audio data can be analyzed. When the audio source includes a valve or a water pipe, the change trend of the slippery stain can be determined according to the audio data. For example, when the audio source of the audio data includes a valve, if it is detected through the audio data that the valve emits a sound of leaking liquid, such as the sound of liquid dripping, flowing, etc., then the change trend of the slippery stain can be an increase.

[0070] In the embodiments of the present application, by determining the change trend of the slippery stain through the multi-modal data at at least one acquisition moment, the slippery stain in a flowing state can be monitored, improving the accuracy of determining the stain state.

[0071] In some embodiments, when generating the risk assessment information of the target object, the following steps can be performed:

[0072] Step S11: Generate the risk assessment information of the target object based on the movement trend of the target object and the change trend of the slippery stain; the risk assessment information is used to characterize the probability that the target object passes through the slippery stain in the changing process.

[0073] Here, since the changing trend of the slippery stain can determine the state of the slippery stain over a period of time, therefore, based on the movement trend of the target object and the changing trend of the slippery stain, the risk assessment information of the target object generated can characterize the probability that the target object passes through the slippery stain in the process of change, so as to judge whether the flow direction of the slippery stain and the action trajectory of the target object coincide at a future time point.

[0074] In some embodiments, the multi-modal data includes at least visual images. The attribute information of the slippery stain can be determined based on the visual images at at least one acquisition moment, and then the risk assessment information of the target object can be generated based on the attribute information of the slippery stain, the movement trend of the target object, and the changing trend of the slippery stain.

[0075] In some embodiments, the action trajectory of the target object can be predicted through the movement trend of the target object, and then the risk assessment information of the target object can be generated based on the action trajectory of the target object and the changing trend of the slippery stain. In implementation, the action trajectory of the target object can be predicted by using a motion model. The motion model can include but is not limited to: Kalman filter model, particle filter model, etc.

[0076] In some embodiments, based on the changing trend of the slippery stain, a geometric region can be modeled for the region where the slippery stain may extend, and then the intersection probability between the action trajectory of the target object and the geometric region can be calculated, so as to generate the risk assessment information of the target object. In some embodiments, the risk assessment information can be represented in any form including but not limited to: percentages, decimals, etc. For example, the risk assessment information can be 81%, 0.6, etc. In some embodiments, the value of the risk assessment information is within a certain range. For example, the value of the risk assessment information can be within [0, 1].

[0077] In the embodiments of the present application, by the movement trend of the target object and the changing trend of the slippery stain, the risk assessment information of the target object is generated, which can determine the probability that the target object passes through the slippery stain in the process of change, and improves the accuracy of generating the risk assessment information.

[0078] In some embodiments, the multi-modal data includes at least visual images. When generating the risk assessment information of the target object, the following steps can be performed:

[0079] Step S111, determine the attribute information of the slippery stain based on the visual images at at least one acquisition moment; the attribute information of the slippery stain includes at least one of the following: slippery stain type, slippery stain depth.

[0080] Here, the visual image at at least one acquisition moment can be an image captured using a camera. The camera can include, but is not limited to, at least one of a high-definition camera, a binocular camera, a depth camera, etc.

[0081] The attribute information of the slippery stain refers to various data and metrics that describe the slippery stain. In some embodiments, the attribute information of the slippery stain can include, but is not limited to, at least one of the slippery stain type, the slippery stain depth, the friction coefficient of the surface of the slippery stain, etc. The slippery stain type can include, but is not limited to, water stains, oil stains, etc. In some embodiments, the slippery stain depth can be expressed in any unit including, but not limited to, centimeters, millimeters, etc. For example, the slippery stain depth can be 0.5 cm (centimeters), 35 mm (millimeters), etc. In some embodiments, for slippery stains of the same area, the slippery stain depth can be the same or different.

[0082] In some embodiments, the friction coefficients of the surfaces of different types of slippery stains can be different. The friction coefficient can be used to evaluate the slipperiness of the surface of the slippery stain. In some embodiments, the higher the friction coefficient, the lower the slipperiness of the slippery stain. Correspondingly, the lower the friction coefficient, the higher the slipperiness of the slippery stain.

[0083] In some embodiments, the visual image can be analyzed through image processing and analysis techniques to determine the color, texture, and shape of the slippery stain, and then the slippery stain type can be determined based on the color, texture, and shape of the slippery stain.

[0084] In some embodiments, the visual image includes visual information, and the slippery stain depth can be determined through the visual information of the visual image. Alternatively, by training a machine learning model, the robot can be enabled to have the function of determining the slippery stain depth from the visual image.

[0085] Step S112, generate risk assessment information for the target object based on the attribute information of the slippery stain, the movement trend of the target object, and the change trend of the slippery stain.

[0086] Here, for the slippery stains with the movement trend of the same target object and the change trend of the slippery stain, when the attribute information of the slippery stain is different, the risk assessment information of the target object may be different. Therefore, it is necessary to jointly generate the risk assessment information for the target object based on the attribute information of the slippery stain, the movement trend of the target object, and the change trend of the slippery stain.

[0087] Exemplarily, for the slippery stains with the movement trend of the same target object and the change trend of the slippery stain, the friction coefficients of the surfaces of different types of slippery stains may be different. Therefore, the attribute information of the slippery stain will affect the risk assessment information.

[0088] In some embodiments, based on the change trend of the slippery stain and the attribute information of the slippery stain, a geometric region can be modeled for the area where the slippery stain may extend, and then the movement trajectory of the target object can be predicted through the movement trend of the target object, so that the intersection probability between the movement trajectory of the target object and the geometric region can be calculated to generate the risk assessment information of the target object.

[0089] In the embodiments of the present application, the risk assessment information of the target object is generated through the attribute information of the slippery stain, the movement trend of the target object, and the change trend of the slippery stain. Considering the influence of different depths of stains with the same area on the risk assessment information, the accuracy of generating the risk assessment information is improved.

[0090] In some embodiments, the stain state includes the target stain area where the slippery stain is located, and the multi-modal data includes infrared images and visual images. When determining the stain state corresponding to the slippery stain in the target scene, the following steps can be performed:

[0091] Step S123, determine the first stain area based on the infrared image and determine the second stain area based on the visual image.

[0092] Here, the first stain area refers to the stain area where the slippery stain is located in the infrared image. The second stain area refers to the stain area where the slippery stain is located in the visual image. In some embodiments, the first stain area and the second stain area may be the same area or different areas.

[0093] The method for determining the first stain area may include but is not limited to: determining the first stain area through the temperature difference of the infrared image, determining the first stain area through the texture of the infrared image, etc. For example, the slippery stain may have a temperature difference from the surrounding environment due to evaporation or the heat absorption characteristics of the material, and the area in the infrared image that appears as a high brightness or a change in the thermal radiation value can be used as the first stain area. Another example is that the stain area may exhibit different texture features due to material changes. The texture features can be extracted from the infrared image using the gray-level co-occurrence matrix, and then classifiers such as support vector machines and random forests can be used to separate the first stain area from the infrared image.

[0094] The method for determining the second stain area may include but is not limited to: determining the second stain area through the gradient change of the visual image, determining the second stain area through the color space conversion of the visual image, etc. For example, the gradient of the visual image in the x direction and the y direction can be calculated through the Sobel operator, and then the contour detection method can be used to take the area with obvious gradient changes as the second stain area. Another example is that the visual image can be converted from the RGB color space to the Lab or YUV color space, etc., and then the second stain area can be separated from the visual image using the color channels.

[0095] In some embodiments, when determining the stain areas (including the first stain area and the second stain area) based on images (including infrared images and visual images), the images can be preprocessed and feature extracted first to obtain information such as the edge information, texture features, and color features of the images, thereby improving the accuracy of the stain areas. The preprocessing can include, but is not limited to: denoising, grayscale conversion, enhancement, etc.

[0096] Step S124, based on the first stain area and the second stain area, determine the target stain area.

[0097] Here, the target stain area refers to the area where the slippery stain is located in the target scene. In some embodiments, the target stain area, the first stain area, and the second stain area can be the same area or different areas.

[0098] In some embodiments, the area with a larger area can be selected from the first stain area and the second stain area as the target stain area, thereby determining the target stain area.

[0099] In some embodiments, the intersection or union area of the first stain area and the second stain area can be determined, and then the intersection or union area can be used as the target stain area, thereby determining the target stain area. During implementation, the intersection and union areas of the first stain area and the second stain area can be calculated by calculating the intersection-over-union ratio of the first stain area and the second stain area.

[0100] In some embodiments, there may be no overlapping relationship between the first stain area and the second stain area at all. The merged area corresponding to the first stain area and the second stain area can be determined, and then the merged area can be used as the target stain area. During implementation, the merged area corresponding to the first stain area and the second stain area can be determined through a region merging algorithm or a merging tool. The region merging algorithm can include, but is not limited to: region growing method, region merging method, region splitting and merging method.

[0101] In the embodiments of the present application, by determining the target stain area from the first stain area determined by the infrared image and the second stain area determined by the visual image, multi-dimensional perception of the slippery stain can be combined with the temperature imaging of the infrared and the visual object recognition, improving the accuracy of determining the stain state.

[0102] In some embodiments, when generating the risk assessment information of the target object, the following steps can be performed:

[0103] Step S12, based on the movement trend of the target object and the target stain area, determine the first probability that the target object passes through the target stain area and the second probability that the target object contacts the target stain area.

[0104] Here, since the target stain area is evaluated with a fixed slippery stain, it is necessary to determine whether the target object will pass through and come into contact with the slippery stain, and thus it is necessary to determine the first probability and the second probability.

[0105] The first probability refers to the probability that the target object passes through the target stain area. The second probability refers to the probability that the target object comes into contact with the target stain area. In some embodiments, the magnitudes of the first probability and the second probability may be the same or different. In some embodiments, the first probability and the second probability may be expressed in any form including but not limited to: percentages, decimals, etc. For example, the first probability may be 50%, 0.9, etc. The second probability may be 33.3%, 0.7, etc. In some embodiments, the values of the first probability and the second probability are within a certain range. For example, the values of the first probability and the second probability may be within [0, 1].

[0106] In some embodiments, the action trajectory of the target object can be predicted based on the movement trend of the target object, and then based on the action trajectory of the target object and the target stain area, the intersection probability of the action trajectory of the target object and the target stain area is calculated, and thus this intersection probability is used as the first probability to determine the first probability. During implementation, the action trajectory of the target object can be predicted by using a motion model. The motion model may include but not limited to: Kalman filter model, particle filter model, etc.

[0107] In some embodiments, since the target object will not come into contact with the target stain area if it does not pass through the target stain area, therefore, only when the first probability satisfies being greater than a preset value, is it necessary to determine the second probability. In some embodiments, the preset value may be expressed in any form including but not limited to: percentages, decimals, etc. For example, the preset value may be 0.05, 10%, etc. In some embodiments, the value of the preset value is within a certain range. For example, the value of the preset value may be within [0, 1].

[0108] In some embodiments, the step length and the movement speed of the target object can be determined based on the movement trend of the target object, and then the second probability is determined according to the step length of the target object, the movement speed of the target object, and the target stain area. During implementation, the intersection area of the step length coverage area of the target object at the movement speed of the target object and the target stain area can be calculated, and then the second probability is calculated according to the ratio of the intersection area to the step length coverage area.

[0109] Step S13, generate risk assessment information of the target object based on the first probability and the second probability.

[0110] Here, the accuracy of the risk assessment information determined only by the first probability or the second probability is relatively low. Therefore, it is necessary to generate the risk assessment information of the target object based on the first probability and the second probability.

[0111] In some embodiments, the target probability can be determined based on the first probability and the second probability, and then the target probability can be used as the risk assessment information of the target object. The target probability can be the larger one of the first probability and the second probability, or the weighted value corresponding to the first probability and the second probability, or the product corresponding to the first probability and the second probability. The present application does not limit the method for determining the target probability.

[0112] In some embodiments, the risk assessment information can be the conditional probability of the first probability and the second probability. In implementation, it can be assumed that the existence of a safety risk for the target object depends on contacting the target stain area, and contacting the stain area depends on passing through the stain area, so that the risk assessment information of the target object can be generated.

[0113] In the embodiments of the present application, the risk assessment information of the target object is generated based on the first probability and the second probability determined by the movement trend of the target object and the target stain area. Considering different situations where the target object passes through and contacts slippery stains, the accuracy of generating the risk assessment information is improved, making the generated risk assessment information closer to the actual usage scenario.

[0114] In some embodiments, when generating the target policy to be executed, the following steps can be performed:

[0115] Step S141, when the movement trend indicates that the target object is in a stationary state, the generated first preset policy is used as the target policy; wherein, the first preset policy includes at least one of the following: the closing method corresponding to the target valve, the wiping strategy corresponding to the slippery stain.

[0116] Here, since the target object does not generate an action trajectory when it is in a stationary state, the slippery stain can be preferentially processed, so that the generated first preset policy can be used as the target policy.

[0117] The first preset policy is used to process the slippery stain. The first preset policy can include at least one of the following: the closing method corresponding to the target valve, the wiping strategy corresponding to the slippery stain, etc. The target valve refers to a valve with a leakage situation. In some embodiments, the target valve in the open state may have a leakage situation. The wiping strategy can include but is not limited to: local wiping, wiping along the line, comprehensive wiping, zonal wiping, etc.

[0118] In some embodiments, the robot can carry devices such as robotic arms, so that when the first preset strategy is the target strategy, specific task operations can be performed through the robotic arm. For example, when the target strategy is to close the target valve, the robot can accurately locate and close the target valve through the robotic arm. For another example, when the target strategy is to wipe slippery stains, the robot can use the cleaning tool carried by the robotic arm to accurately find and wipe the slippery stains.

[0119] In some embodiments, when the first preset strategy includes a closing method corresponding to the target valve, the attribute information of the target valve can be determined through the target model based on multi-modal data, and then, based on the attribute information of the target valve, the closing method corresponding to the target valve can be determined.

[0120] In some embodiments, when the first preset strategy includes a wiping strategy corresponding to the slippery stain, the geometric features corresponding to the slippery stain can be determined first, and then, based on the geometric features corresponding to the slippery stain, the distribution information of the slippery stain can be determined, so that, based on the distribution information of the slippery stain, the wiping strategy corresponding to the slippery stain can be determined.

[0121] In some embodiments, when the change trend of the slippery stain indicates that the slippery stain continues to grow, the closing method corresponding to the target valve is used as the target strategy, so as to control the robot to switch the target valve to the closed state based on the target strategy.

[0122] Step S142, when the motion trend indicates that the target object is in a moving state, determine the activity area of the target object, so as to use the generated second preset strategy as the target strategy based on the activity area of the target object and the change trend of the slippery stain; wherein, the second preset strategy is used to remind the target object to adjust the motion mode.

[0123] Here, since the target object will generate an action trajectory when the target object is in a moving state, it is necessary to determine the activity area of the target object, so as to use the generated second preset strategy as the target strategy based on the activity area of the target object and the change trend of the slippery stain.

[0124] The activity area of the target object refers to the range where the target object conducts activities. The activity area of the target object can be of any shape, for example, circular, rectangular, polygonal, etc.

[0125] In some embodiments, the activity area of the target object can be determined through the action trajectory of the target object. By analyzing the action trajectory of the target object, determine the area covered by the action trajectory of the target object and the possible moving directions of the target object, so as to delimit a geographical or spatial range as the activity area of the target object.

[0126] In some embodiments, the target duration required for the robot to execute the first preset policy can be determined, and then based on the moving speed of the target object and the target duration, the activity area of the target object can be determined.

[0127] The second preset policy is used to remind the target object to adjust its movement mode. In some embodiments, the robot can be equipped with a speaker, so that when the second preset policy is the target policy, a voice can be emitted through the speaker to remind the target object to adjust its movement mode.

[0128] In some embodiments, in response to the completion of the execution of the second preset policy, the movement trend of the target object can be determined, so as to generate a target policy to be executed based on the movement trend of the target object and the stain state.

[0129] In the embodiments of the present application, when the movement trend indicates that the target object is in a stationary state or a moving state, different processes are adopted to generate the target policy to be executed, which can determine the corresponding target policy according to different actual scenarios, improving the accuracy of generating the target policy while also improving the compatibility of the control method.

[0130] In some embodiments, when the slippery stain continues to grow, the following steps can be executed:

[0131] Step S105, when the change trend of the slippery stain indicates that the slippery stain continues to grow, the closing method corresponding to the target valve is used as the target policy, and based on the target policy, the robot is controlled to switch the target valve to the closed state.

[0132] Here, since the continuous growth of the slippery stain can indicate that there may be a leakage situation in the target valve, therefore, it is necessary to preferentially use the closing method corresponding to the target valve as the target policy and execute the target policy.

[0133] In some embodiments, multi-modal data at at least one acquisition moment can be determined, and then based on the multi-modal data at at least one acquisition moment, the change trend of the slippery stain can be determined.

[0134] In some embodiments, the robot can be equipped with devices such as a robotic arm, so that when the closing method corresponding to the target valve is the target policy, the robotic arm is used to accurately position and close the target valve, and based on the target policy, the robot is controlled to switch the target valve to the closed state.

[0135] In the embodiments of the present application, when the changing trend of the slippery stain indicates continuous growth of the slippery stain, the closing method corresponding to the target valve is used as the target strategy to switch the target valve to the closed state. Only when the slippery stain increases is it determined that a leakage occurs, and then the target valve is preferentially closed, which can reduce the losses caused by continuous leakage. At the same time, since closing the target valve can suppress the leakage situation, the safety risk of the target object is reduced.

[0136] In some embodiments, when determining the activity area of the target object, the following steps may be performed:

[0137] Step S1421, determine the target duration required for the robot to execute the first preset strategy.

[0138] Here, the target duration refers to the duration required for the robot to execute the first preset strategy. In some embodiments, the target duration can be expressed in any unit including but not limited to: seconds, minutes, etc. For example, the target duration can be 30s (seconds), 1.3min (minutes), etc.

[0139] The method for determining the target duration may include but not be limited to: determining the target duration based on historical execution records, determining the target duration based on simulation, etc. For example, the historical execution records of the robot executing the strategy can be obtained, multiple target execution records of executing the first preset strategy are selected from the historical execution records, and then the target duration is determined according to the duration of the target execution records. For another example, a simulation environment model of the target scenario can be established, and then the execution process of the first preset strategy is simulated in the simulation environment, and the time required to execute the first preset strategy is recorded, so as to determine the target duration.

[0140] In some embodiments, when determining the target duration, it is necessary to calculate the duration required to execute all the first preset strategies. For example, when the first preset strategy includes the closing method corresponding to the target valve and the wiping strategy corresponding to the slippery stain, the target duration is the sum of the duration required to execute the closing method corresponding to the target valve and the duration required to execute the wiping strategy corresponding to the slippery stain.

[0141] Step S1422, determine the activity area of the target object based on the moving speed of the target object and the target duration.

[0142] Here, the activity area of the target object can be of any shape, for example, circular, rectangular, polygonal, etc. In some embodiments, the value of the moving speed of the target object is within a certain range. For example, the value of the moving speed of the target object can be within [0, 5], and the moving speed of the target object can be 1m / s (meters per second), 1.3m / s, etc.

[0143] In some embodiments, the movement trajectory of the target object can be planned based on the movement speed and target duration of the target object. Then, by analyzing the movement trajectory of the target object, the area covered by the movement trajectory of the target object and the possible movement directions of the target object can be determined, so as to delimit a geographical or spatial range as the activity area of the target object.

[0144] In the embodiments of the present application, the activity area of the target object is determined by the movement speed and target duration of the target object, which improves the accuracy of determining the activity area of the target object.

[0145] In some embodiments, when generating the target policy to be executed, the following steps can be performed:

[0146] Step S106, in response to the completion of the execution of the second preset policy, determine the movement trend of the target object, so as to generate the target policy to be executed based on the movement trend of the target object and the stain state.

[0147] Here, after reminding the target object to adjust the movement mode based on the second preset policy, the movement trend of the target object can be determined again, so as to regenerate the target policy to be executed based on the movement trend of the target object and the stain state.

[0148] In some embodiments, when the movement trend indicates that the target object is in a stationary state, the generated first preset policy is used as the target policy. When the movement trend indicates that the target object is in a moving state, determine the activity area of the target object, so as to use the generated second preset policy as the target policy based on the activity area of the target object and the change trend of the slippery stain.

[0149] In the embodiments of the present application, after the execution of the second preset policy is completed, the movement trend of the target object can be determined again and the target policy to be executed can be regenerated, which can generate a target policy corresponding to the current actual situation, making the target policy have real-time performance and improving the accuracy of generating the target policy.

[0150] In some embodiments, when the first preset policy includes the closing method corresponding to the target valve, the following steps can be performed:

[0151] Step S107, based on the multimodal data, determine the attribute information of the target valve through the target model.

[0152] Here, the attribute information of the target valve may include but is not limited to at least one of the name of the target valve, the position of the target valve, the operation method of the target valve, etc.

[0153] The target model refers to a model with the function of processing multimodal data. The target model can be of any suitable type, for example, a deep learning model, a generative artificial intelligence (Artificial Intelligence Generated Content, AIGC) model, etc. Deep learning is to learn the internal laws and representation levels of sample data. By using a suitable deep learning model, the attribute information of the target valve can be determined. Deep learning models include but are not limited to convolutional neural networks, recurrent neural networks, etc. AIGC refers to the technical method of artificial intelligence based on large pre-trained models, etc., which generates relevant content through the learning and recognition of existing data with appropriate generalization ability.

[0154] In some embodiments, the untrained target model can be trained through the attribute information of multiple valves and multimodal data, so as to obtain the target model. At the same time, the target model can also be updated according to the newly obtained attribute information of the valve and multimodal data, thereby improving the accuracy of the target model.

[0155] Step S108, based on the attribute information of the target valve, determine the closing method corresponding to the target valve.

[0156] Here, through the closing method corresponding to the target valve, the robot can switch the target valve to the closed state. In some embodiments, the closing methods corresponding to different valves may be the same or different.

[0157] In some embodiments, based on the attribute information of the target valve, a natural language processing model or a multimodal model can be used to determine the closing method corresponding to the target valve. For example, the natural language processing model can, based on the attribute information of the target valve, combine the text description of the valve to more comprehensively understand the characteristics of the valve, thereby determining the closing method corresponding to the target valve.

[0158] In some embodiments, through a target detection algorithm, based on the attribute information of the target valve, the closing method corresponding to the target valve can be determined. Target detection algorithms include but are not limited to: Fully Convolutional One-Stage Object Detection (FCOS) algorithm, Single Shot MultiBox Detector (SSD) algorithm, etc. The FCOS algorithm is similar to the semantic segmentation method and can perform multi-scale prediction on the pixel points of image data. The SSD algorithm is a single-stage target detection algorithm that extracts features through a convolutional neural network and then extracts different feature layers for detection output.

[0159] In some embodiments, a second correspondence relationship can be established between the attribute information of the valve and the closing method. After determining the attribute information of the target valve, the corresponding closing method for the target valve can be determined through the second correspondence relationship.

[0160] In the embodiments of the present application, the attribute information of the target valve is determined through multimodal data and the target model, so that based on the attribute information of the target valve, the corresponding closing method for the target valve can be determined. The corresponding closing method can be selected according to different valves, improving the accuracy of generating the first preset strategy.

[0161] In some embodiments, when the target valve has not been switched to the closed state, the following steps can be executed:

[0162] Step S109, in response to the target valve not being switched to the closed state, trigger the execution of the third preset strategy; wherein, the third preset strategy includes at least one of the following: locating the water leakage position, analyzing the cause of water leakage, and the processing strategy for the target object.

[0163] Here, since the target valve not being switched to the closed state may indicate that the water leakage situation has not been resolved, it is necessary to trigger the execution of the third preset strategy. The third preset strategy is used to monitor the water leakage state. The third preset strategy includes at least one of the following: locating the water leakage position, analyzing the cause of water leakage, and the processing strategy for the target object.

[0164] In some embodiments, the third preset strategy can be executed in any form, for example, algorithms, models, neural networks, etc. Algorithms, models, neural networks, etc. with functions such as locating the water leakage position and analyzing the cause of water leakage can be used to execute the third preset strategy, and the present application does not limit the execution form of the third preset strategy.

[0165] In some embodiments, when the third preset strategy includes at least one sub-strategy, the execution order of the sub-strategy can be determined according to the acquired multimodal data, and then the third preset strategy is executed. For example, the third preset strategy includes analyzing the cause of water leakage and the processing strategy for the target object. By analyzing the multimodal data, it is found that the target object is about to be contaminated by slippery stains. Therefore, it is necessary to execute the processing strategy for the target object first and then analyze the cause of water leakage.

[0166] In some embodiments, the target object is used to affect the change trend of slippery stains. The processing strategy for the target object includes moving the target object. At least one object passed through during the change process of the slippery stains can be used as the target object, and then the target object is moved based on the order in which the slippery stains pass through the target object. Furthermore, based on the moved target object, the change trend of the slippery stains is determined.

[0167] In some embodiments, the processing strategy for the target object includes moving the target object and disconnecting the power supply of the target object. The distance between an object located in the target scene and the slippery stain can be detected, and at least one object whose distance from the slippery stain is less than a preset distance threshold is taken as the target object. Then, according to the distance between the target object and the slippery stain, the robot is controlled to perform power-off processing and / or moving processing on the target object.

[0168] In the embodiments of the present application, when the target valve is not switched to the closed state, the execution of the third preset strategy is triggered, which can prevent the risks that may be brought by water leakage before the leakage problem is solved, and improve the flexibility and accuracy of the control method.

[0169] In some embodiments, the target object is used to affect the change trend of the slippery stain. The processing strategy for the target object includes moving the target object. When determining the target object, step S110a can be executed. When the execution of the third preset strategy is triggered, step S191 and step S192 can be executed, where:

[0170] Step S110a: Take at least one object that passes through during the change process of the slippery stain as the target object.

[0171] Here, since the target object may come into contact with the slippery stain during the change process, the target object can be used to affect the change trend of the slippery stain, and thus the change trend of the slippery stain can be affected by moving the target object.

[0172] In some embodiments, multi-modal data at at least one acquisition moment can be determined, and then based on the multi-modal data at at least one acquisition moment, the change trend of the slippery stain can be determined, so as to determine the change process of the slippery stain according to the change trend of the slippery stain.

[0173] In some embodiments, the change area of the slippery stain can be determined according to the change process of the slippery stain, and then the objects within the change area of the slippery stain are taken as at least one object that passes through during the change process of the slippery stain to determine the target object.

[0174] Step S191: Move the target object based on the order in which the slippery stain passes through the target object.

[0175] Here, the target object can be moved in the order in which the slippery stain passes through the target object from early to late.

[0176] In some embodiments, the robot can be equipped with devices such as a robotic arm and a mobile platform, and the robotic arm can move the target object based on the order in which the slippery stain passes through the target object.

[0177] In some embodiments, when moving a target object, it is necessary to consider the attribute information of the target object, and determine the corresponding moving method of the target object according to the attribute information of the target object. The attribute information of the target object may include, but is not limited to, at least one of the size of the target object, the shape of the target object, the material of the target object, etc.

[0178] Step S192, based on the moved target object, determine the change trend of the slippery stain.

[0179] Here, since moving the target object can affect the change process of the slippery stain, it is necessary to re-determine the change trend of the slippery stain.

[0180] In some embodiments, multi-modal data at at least one acquisition moment can be determined, and then based on the multi-modal data at at least one acquisition moment, the change trend of the slippery stain can be determined.

[0181] In some embodiments, after determining the change trend of the slippery stain, the change process of the slippery stain can be re-determined, and then at least one object passed through during the change process of the slippery stain can be used as the target object, so as to move the target object again according to the re-determined target object.

[0182] In the embodiments of the present application, at least one object passed through during the change process of the slippery stain is used as the target object, so as to move the target object based on the order in which the slippery stain passes through the target object, and determine the change trend of the slippery stain based on the moved target object, which can affect the flow direction of the slippery stain by moving the target object, reduce the possibility of the target object coming into contact with the slippery stain, and reduce the safety risk of the target object.

[0183] In some embodiments, the processing strategy of the target object includes moving the target object and disconnecting the power supply of the target object. When determining the target object, step S110b can be executed. When triggering the execution of the third preset strategy, step S193 can be executed, where:

[0184] Step S110b, detect the distance between the object located in the target scene and the slippery stain, and use at least one object whose distance from the slippery stain is less than the preset distance threshold as the target object.

[0185] Here, the preset distance threshold can be expressed in any unit including, but not limited to, meters, centimeters, etc. For example, the preset distance threshold can be 3m, 90cm, etc. Exemplarily, the distance between a certain object and the slippery stain is 2m, and the preset distance threshold is 3m, which meets the condition that the distance from the slippery stain is less than the preset distance threshold. Therefore, this object can be used as the target object.

[0186] In some embodiments, the robot can be equipped with a depth camera to capture a depth image of the target scene using the depth camera. Then, based on the depth image, the Euclidean distance between the center point of the object and the edge or center point of the slippery stain area can be calculated, so that the distance between the object located in the target scene and the slippery stain can be detected.

[0187] In some embodiments, the robot can be equipped with a lidar to scan the target scene using the lidar to generate a three-dimensional point cloud of the objects in the target scene. Then, based on the three-dimensional point cloud of the object, the distance between the object located in the target scene and the slippery stain can be detected.

[0188] Step S193, control the robot to perform a power-off process and / or a movement process on the target object according to the distance between the target object and the slippery stain.

[0189] Here, since the target object may include an object connected to a power source, therefore, it is necessary to first perform a power-off process on the target object and then perform a movement process.

[0190] In some embodiments, according to the distance between the target object and the slippery stain, the robot can be controlled to perform a power-off process and / or a movement process on the target object in ascending order. For the target object connected to a power source, first perform a power-off process and then a movement process. For the target object not connected to a power source, the movement process can be directly performed.

[0191] In the embodiments of the present application, at least one object whose distance from the slippery stain is less than a preset distance threshold is used as the target object. Thus, according to the distance between the target object and the slippery stain, the robot is controlled to perform a power-off process and / or a movement process on the target object, which can reduce the possibility of electric leakage of the object by moving and powering off the target object, thereby reducing the safety risk of the target object and reducing the possibility of object damage at the same time.

[0192] In some embodiments, when determining the wiping strategy corresponding to the slippery stain, the following steps can be performed:

[0193] Step S1411, determine the geometric features corresponding to the slippery stain.

[0194] Here, the geometric features corresponding to the slippery stain can include but are not limited to at least one of the area of the slippery stain, the perimeter of the slippery stain, the center point of the slippery stain, the shape of the slippery stain, etc.

[0195] The method for determining the geometric features corresponding to slippery stains may include, but is not limited to: determining the geometric features corresponding to slippery stains through edge detection, determining the geometric features corresponding to slippery stains through feature extraction, etc. For example, multi-modal data can be analyzed through edge detection to detect the edges of slippery stains in the multi-modal data, and then the geometric features corresponding to the slippery stains can be determined based on the edges of the slippery stains. Among them, edge detection can be implemented by means of Sobel operators, Canny algorithms, etc. Also for example, edge information, texture features, color features, etc. of the multi-modal data are obtained through feature extraction, and then the geometric features corresponding to the slippery stains are determined based on the extracted information.

[0196] Step S1412: Based on the geometric features corresponding to the slippery stains, determine the distribution information of the slippery stains.

[0197] Here, the distribution information of the slippery stains may include, but is not limited to: at least one of dot distribution, line distribution, surface distribution, etc.

[0198] In some embodiments, the liquid area of the slippery stains can be determined according to the geometric features corresponding to the slippery stains, and then the distribution information of the slippery stains can be determined based on the liquid area of the slippery stains. For example, if the area of the liquid area of the slippery stains is small and the distribution of the slippery stains is relatively discrete, it can be determined that the distribution information of the slippery stains is dot distribution. Also for example, if the liquid area of the slippery stains presents obvious linear features, such as extending along a certain direction, it can be determined that the distribution information of the slippery stains is line distribution.

[0199] In some embodiments, a third correspondence relationship can be established between the geometric features and the distribution information, so that after determining the geometric features corresponding to the slippery stains, the distribution information of the slippery stains can be determined through this third correspondence relationship.

[0200] Step S1413: Based on the distribution information of the slippery stains, determine the wiping strategy corresponding to the slippery stains.

[0201] Here, the wiping strategy corresponding to the slippery stains may include, but is not limited to: at least one of local wiping, wiping along a line, overall wiping, zonal wiping, etc.

[0202] In some embodiments, the wiping strategies corresponding to different distribution information may be the same or different. At the same time, since the distribution information of the slippery stains may be at least one kind, different wiping strategies can be adopted for different distribution information.

[0203] In some embodiments, a fourth correspondence relationship can be established between the distribution information and the wiping strategy, so that after determining the distribution information of the slippery stain, the wiping strategy corresponding to the slippery stain can be determined through this fourth correspondence relationship. For example, when the distribution information of the slippery stain is linear distribution, the wiping strategy corresponding to the slippery stain can be wiping along the line; when the distribution information of the slippery stain is planar distribution, the wiping strategy corresponding to the slippery stain can be overall wiping.

[0204] In the embodiments of the present application, the distribution information of the slippery stain is determined by the geometric features corresponding to the slippery stain, so as to determine the wiping strategy corresponding to the slippery stain based on the distribution information of the slippery stain. It is possible to select the corresponding wiping strategy according to the slippery stains with different distributions, thereby improving the accuracy of generating the first preset strategy.

[0205] The following describes the application of the control method provided in the embodiments of the present application in an actual scenario, taking the slippery stain as water stain as an example.

[0206] Water leakage at home will cause huge economic losses. When water stains appear at home, the elderly may slip after contacting the water stains, posing a relatively high safety risk.

[0207] The embodiments of the present application provide a control method. By obtaining multi-modal data, the stain state corresponding to the slippery stain in the target scenario is determined, and then based on the movement trend of the target object and the stain state corresponding to the slippery stain, risk assessment information of the target object is generated. Thus, when the risk assessment information indicates that the slippery stain causes a safety risk to the target object, a target strategy to be executed is generated based on the movement trend of the target object and the stain state. On the one hand, compared with determining the stain state through single-modal data, since the stain state is determined through multi-modal data, the misjudgment probability is reduced while the accuracy of determining the stain state is improved; on the other hand, since the risk assessment information is related to the movement trend and stain state of the target object, corresponding risk assessment information can be generated according to different actual application scenarios, making the generated risk assessment information more flexible and closer to the current application scenario; on the other hand, since the target strategy to be executed is generated based on the movement trend and stain state, the corresponding target strategy can be selected in real time according to different actual scenarios, improving the decision-making ability and intelligent level of the robot. At the same time, by executing the target strategy, the safety risk of the target object can be reduced, improving the safety of the target object.

[0208] Figure 2 Schematic diagram of the implementation process of a control method provided in the embodiments of the present application Figure 2 , as Figure 2 shown, this method is applied to a home companion robot and includes steps S201 to S206, where:

[0209] In step S201, the robot obtains a visual image of the water stain through a visual camera and an infrared image of the water stain through an infrared camera;

[0210] In step S202, fluid detection is implemented using the visual image and the infrared image to determine the change trend of the water stain;

[0211] In step S203, after detecting a water leakage situation, a voice reminder or a voice alarm can be issued to prompt the user (corresponding to the aforementioned target object) to close the leaking valve (corresponding to the aforementioned target valve), or, based on the specifications of the valve, an appropriate operation method is adapted to close the leaking valve;

[0212] In step S204, when the user does not respond or is unable to respond for a long time and the robot itself is unable to close the valve, help is sought by means such as making a phone call;

[0213] In step S205, the robot automatically generates a wiping strategy based on the range presented by the water stain and wipes the water stain;

[0214] In step S206, before the water leakage situation is resolved, the water leakage state can be continuously monitored, the flow direction of the water leakage can be analyzed, etc., and risks that may be brought about by the water leakage can be prevented.

[0215] In some embodiments, the robot can accompany the target object in real time. When a water stain appears in the area where the target object arrives, the robot can give a timely reminder and intervention to prevent accidents.

[0216] In some embodiments, the robot can infer the position and operation method of valves such as water valves through the capabilities of the graphic and text large model, so that the water valve / switch can be operated by hand or a special fixture according to the operation method to prevent accidents from continuing.

[0217] Based on the above embodiments, the embodiment of the present application further provides a control device, Figure 3 As shown in the composition structure diagram of a control device provided by the embodiment of the present application, Figure 3 As shown, the control device 300 includes an acquisition unit 301, a determination unit 302, a first generation unit 303, and a second generation unit 304, where:

[0218] The acquisition unit 301 is configured to acquire multimodal data corresponding to a target scene; the target scene is the scene where the robot is located during the process of following the target object;

[0219] The determination unit 302 is configured to determine the stain state corresponding to the slippery stain in the target scene based on the multimodal data;

[0220] The first generation unit 303 is configured to generate risk assessment information of the target object based on the movement trend of the target object and the stain state corresponding to the slippery stain.

[0221] The second generation unit 304 is configured to generate a target strategy to be executed based on the movement trend of the target object and the stain state when the risk assessment information indicates that the slippery stain causes a safety risk to the target object.

[0222] In some embodiments, the stain state includes the change trend of the slippery stain. The determination unit 302 is further configured to determine multi-modal data at at least one acquisition moment; based on the multi-modal data at at least one acquisition moment, determine the change trend of the slippery stain; the change trend is used to determine the attribute of the slippery stain at a certain acquisition moment.

[0223] In some embodiments, the first generation unit 303 is further configured to generate risk assessment information of the target object based on the movement trend of the target object and the change trend of the slippery stain; the risk assessment information is used to represent the probability that the target object passes through the slippery stain in the changing process.

[0224] In some embodiments, the multi-modal data includes at least visual images. The first generation unit 303 is further configured to determine the attribute information of the slippery stain based on the visual images at at least one acquisition moment; the attribute information of the slippery stain includes at least one of the following: the type of the slippery stain, the depth of the slippery stain; based on the attribute information of the slippery stain, the movement trend of the target object, and the change trend of the slippery stain, generate risk assessment information of the target object.

[0225] In some embodiments, the stain state includes the target stain area where the slippery stain is located, and the multi-modal data includes infrared images and visual images; the determination unit 302 is further configured to determine a first stain area based on the infrared images and a second stain area based on the visual images; based on the first stain area and the second stain area, determine the target stain area.

[0226] In some embodiments, the first generation unit 303 is further configured to determine a first probability that the target object passes through the target stain area and a second probability that the target object contacts the target stain area based on the movement trend of the target object and the target stain area; based on the first probability and the second probability, generate risk assessment information of the target object.

[0227] In some embodiments, the second generating unit 304 is further configured to use the generated first preset policy as the target policy when the motion trend indicates that the target object is in a stationary state; wherein, the first preset policy includes at least one of the following: a closing method corresponding to the target valve, a wiping strategy corresponding to the slippery stain; when the motion trend indicates that the target object is in a moving state, determine the activity area of the target object, and based on the activity area of the target object and the change trend of the slippery stain, use the generated second preset policy as the target policy; wherein, the second preset policy is used to remind the target object to adjust its motion mode.

[0228] In some embodiments, the control device 300 further includes an execution unit, configured to use the closing method corresponding to the target valve as the target policy when the change trend of the slippery stain indicates that the slippery stain continues to grow, and based on the target policy, control the robot to switch the target valve to the closed state.

[0229] In some embodiments, the second generating unit 304 is further configured to determine the target duration required for the robot to execute the first preset policy; and determine the activity area of the target object based on the moving speed of the target object and the target duration.

[0230] In some embodiments, the second generating unit 304 is further configured to, in response to the completion of the execution of the second preset policy, determine the motion trend of the target object, and based on the motion trend of the target object and the stain state, generate a target policy to be executed.

[0231] In some embodiments, when the first preset policy includes the closing method corresponding to the target valve, the determining unit 302 is further configured to determine the attribute information of the target valve through the target model based on the multimodal data; and determine the closing method corresponding to the target valve based on the attribute information of the target valve.

[0232] In some embodiments, the execution unit is further configured to, in response to the target valve not being switched to the closed state, trigger the execution of a third preset policy; wherein, the third preset policy includes at least one of the following: locating the water leakage position, analyzing the cause of the water leakage, a processing strategy for the target object.

[0233] In some embodiments, the target object is used to affect the change trend of the slippery stain, and the processing strategy for the target object includes moving the target object. The determining unit 302 is further configured to use at least one object passed through during the change process of the slippery stain as the target object; move the target object based on the order in which the slippery stain passes through the target object; and determine the change trend of the slippery stain based on the moved target object.

[0234] In some embodiments, the processing strategy for the target object includes moving the target object and disconnecting the power supply of the target object. The determining unit 302 is further configured to detect the distance between the object located in the target scene and the slippery stain, and use at least one object whose distance from the slippery stain is less than a preset distance threshold as the target object; and control the robot to perform power-off processing and / or movement processing on the target object according to the distance between the target object and the slippery stain.

[0235] In some embodiments, the determining unit 302 is further configured to determine the geometric features corresponding to the slippery stain; determine the distribution information of the slippery stain based on the geometric features corresponding to the slippery stain; and determine the wiping strategy corresponding to the slippery stain based on the distribution information of the slippery stain.

[0236] Based on the above embodiments, an embodiment of the present application further provides a robot. Figure 4 The following is a schematic diagram of the hardware entity of a robot provided by an embodiment of the present application. As Figure 4 shown, the robot 400 includes a processor 401, a communication interface 402, a memory 403, an image acquisition device 404, and an actuator 405, where:

[0237] The image acquisition device 404 is configured to acquire multi-modal data corresponding to the target scene; the target scene is the scene where the robot is located during the process of following the target object.

[0238] The processor 401 is configured to determine the stain state corresponding to the slippery stain in the target scene based on the multi-modal data; generate risk assessment information of the target object based on the movement trend of the target object and the stain state corresponding to the slippery stain; and generate a target strategy to be executed based on the movement trend of the target object and the stain state when the risk assessment information indicates that the slippery stain causes a safety risk to the target object.

[0239] The actuator 405 is configured to execute the target strategy.

[0240] The processor 401 generally controls the overall operation of the robot 400.

[0241] The communication interface 402 can enable the robot 400 to communicate with other terminals or servers through a network.

[0242] The memory 403 is configured to store instructions and applications executable by the processor 401, and can also cache data to be processed or already processed by the processor 401 and each module in the robot 400 (such as image data, audio data, voice communication data, and video communication data), and can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 401, the communication interface 402, the memory 403, the image acquisition device 404, and the actuator 405 through the bus 406.

[0243] Here, the image acquisition device may include, but is not limited to, at least one of a camera, a radar system, etc. The method for obtaining multimodal data may include, but is not limited to, reading the acquired data from the image acquisition device to obtain multimodal data, receiving the acquired data sent by the image acquisition device to obtain multimodal data, etc.

[0244] The target scene can be any suitable scene, such as indoor, outdoor, etc. The multimodal data may include, but is not limited to, at least one of image data, audio data, etc. The slippery stains may include, but is not limited to, at least one of water stains, oil stains, etc.

[0245] The movement trend of the target object can indicate that the target object is in a stationary state or a moving state. In some embodiments, the risk assessment information can be represented in any form including, but not limited to, percentages, decimals, etc. For example, the risk assessment information can be 81%, 0.6, etc. In some embodiments, the value of the risk assessment information is within a certain range. For example, the value of the risk assessment information can be within [0, 1].

[0246] The target policy to be executed may include, but is not limited to, one of a first preset policy, a second preset policy, a third preset policy, etc. The method for generating the target policy may include, but is not limited to, generating the target policy based on the movement trend and stain state of the target object through machine learning, generating the target policy based on the movement trend and stain state of the target object through multi-objective optimization, etc.

[0247] In the embodiments of the present application, the robot acquires multimodal data through an image acquisition device, and then the processor determines the stain state corresponding to the slippery stain in the target scene based on the multimodal data. Then, based on the movement trend of the target object and the stain state corresponding to the slippery stain, risk assessment information of the target object is generated. Thus, when the risk assessment information indicates that the slippery stain poses a safety risk to the target object, a target strategy to be executed is generated based on the movement trend of the target object and the stain state, so as to execute the target strategy through an actuator. On the one hand, compared with determining the stain state through single-modal data, since the stain state is determined through multimodal data, the probability of misjudgment is reduced while the accuracy of determining the stain state is improved. On the other hand, since the risk assessment information is related to the movement trend of the target object and the stain state, corresponding risk assessment information can be generated according to different actual application scenarios, making the generated risk assessment information more flexible and closer to the current application scenario. On the other hand, since the target strategy to be executed is generated based on the movement trend and the stain state, the corresponding target strategy can be selected in real time according to different actual scenarios, improving the decision-making ability and intelligent level of the robot. At the same time, by executing the target strategy, the safety risk of the target object can be reduced, improving the safety of the target object.

[0248] The description of the above embodiments of the electronic device is similar to the description of the above method embodiments and has similar beneficial effects to the method embodiments. For the technical details not disclosed in the embodiments of the electronic device of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0249] It should be pointed out here that: the above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and their similarities can be referred to each other. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects to the method embodiments. For the technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0250] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of software function modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence or the part that contributes to the related technology, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0251] The embodiments of the present application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the computer program, the above method is implemented.

[0252] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented. The computer-readable storage medium can be transient or non-transient.

[0253] The embodiments of the present application provide a computer program product. The computer program product includes a non-transient computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, part or all of the steps in the above method are implemented. The computer program product can be specifically implemented in a manner of hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0254] It should be pointed out here that: the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments and have beneficial effects similar to those of the method embodiments. For the technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the descriptions of the method embodiments of the present application for understanding.

[0255] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above steps / processes do not mean the order of execution. The order of execution of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0256] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0257] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the couplings, direct couplings or communication connections between the components shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be electrical, mechanical or other forms.

[0258] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] In addition, each functional unit in the embodiments of the present application can be all integrated in a processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in a unit; the above integrated unit can be implemented in the form of hardware, or in the form of a hardware plus a software functional unit.

[0260] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media that can store program codes such as removable storage devices, read-only memories, magnetic disks, or optical discs.

[0261] Alternatively, if the above integrated units are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the related art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. And the foregoing storage medium includes: various media that can store program codes such as removable storage devices, ROMs, magnetic disks, or optical discs.

[0262] As described above, only the implementation manners of the present application are provided, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application.

Claims

1. A control method, characterized in that: Applied to a robot, the method comprises: Acquire multimodal data corresponding to a target scene; the target scene is a scene in which the robot is located during the process of following a target object; Based on the multimodal data, determining a stain state corresponding to a slippery stain in the target scene; generating risk assessment information of the target object based on the movement trend of the target object and the stain state corresponding to the slippery stain; In the case where the risk assessment information indicates that the slippery stain causes a safety risk to the target object, a target strategy to be executed is generated based on the movement trend of the target object and the stain state.

2. The method according to claim 1, characterized in that: The stain state includes a change trend of the slippery stain or a target stain area where the slippery stain is located; and determining the stain state corresponding to the slippery stain in the target scene based on the multimodal data includes one of the following: Determine multimodal data at at least one acquisition moment, and determine a change trend of the slippery stain based on the multimodal data at at least one acquisition moment; The change trend is used to determine the properties of the slippery stain at a certain collection moment; A first stain area is determined based on the infrared image, and a second stain area is determined based on the visual image, so as to determine the target stain area based on the first stain area and the second stain area.

3. The method according to claim 2, characterized in that The generating risk assessment information of the target object based on the movement trend of the target object and the stain state corresponding to the slippery stain includes one of the following: generating risk assessment information of the target object based on the movement trend of the target object and the change trend of the slippery stain; The risk assessment information is used to characterize the probability of the target object passing through the slippery stain in the process of changing; Based on the movement trend of the target object and the target stain area, a first probability that the target object passes through the target stain area and a second probability that the target object contacts the target stain area are determined, so as to generate risk assessment information of the target object based on the first probability and the second probability.

4. The method according to claim 3, characterized in that The multimodal data at least includes a visual image, and the risk assessment information of the target object is generated based on the movement trend of the target object and the change trend of the slippery stain, including: Determine the property information of the slippery stain based on the visual image at the at least one acquisition moment; the property information of the slippery stain includes at least one of the following: the slippery stain type and the slippery stain depth; Based on the attribute information of the slippery stain, the movement trend of the target object and the change trend of the slippery stain, risk assessment information of the target object is generated.

5. The method according to claim 2, characterized in that: The generating a target strategy to be executed based on the movement trend of the target object and the stain state includes one of the following: When the motion trend indicates that the target object is in a stationary state, the generated first preset strategy is used as the target strategy; wherein the first preset strategy includes at least one of the following: a closing method corresponding to the target valve, a wiping strategy corresponding to the slippery stain; When the motion trend indicates that the target object is in motion, the target duration required for the robot to execute the first preset strategy is determined, and the activity area of ​​the target object is determined based on the moving speed of the target object and the target duration, so as to generate a second preset strategy as the target strategy based on the activity area of ​​the target object and the changing trend of the slippery stains; wherein the second preset strategy is used to remind the target object to adjust its motion mode.

6. The method according to claim 5, characterized in that The method further comprises at least one of the following: In a case where the change trend of the slippery stain indicates that the slippery stain continues to grow, the closing mode corresponding to the target valve is used as the target strategy, so as to control the robot to switch the target valve to a closed state based on the target strategy; In response to the completion of the execution of the second preset strategy, determining the movement trend of the target object to generate the target strategy to be executed based on the movement trend of the target object and the stain state; In the case where the first preset strategy includes a closing mode corresponding to a target valve, based on the multimodal data, determining the attribute information of the target valve through a target model, so as to determine the closing mode corresponding to the target valve based on the attribute information of the target valve; In response to the target valve not being switched to the closed state, the execution of a third preset strategy is triggered; wherein the third preset strategy includes at least one of the following: locating the leakage position, analyzing the leakage cause, and a target object processing strategy.

7. The method according to claim 6, characterized in that The target object is used to affect the change trend of the slippery stain, and the processing strategy of the target object includes moving the target object and / or disconnecting the power supply of the target object. The method also includes one of the following: at least one object passing through during the change of the slippery stain is used as the target object; Detecting the distance between an object in the target scene and the slippery stain, and taking at least one object whose distance to the slippery stain is less than a preset distance threshold as the target object; The triggering of the execution of the third preset strategy includes one of the following: Based on the order in which the slippery stain passes through the target object, the target object is moved to determine a change trend of the slippery stain based on the target object after the movement; According to the distance between the target object and the slippery stain, the robot is controlled to perform power-off processing and / or movement processing on the target object.

8. The method according to any one of claims 5 to 7, characterized in that The method further comprises: Determining geometric features corresponding to the slippery stain; Determining distribution information of the slippery stain based on the geometric features corresponding to the slippery stain; Based on the distribution information of the slippery stain, a wiping strategy corresponding to the slippery stain is determined.

9. A control device, characterized in that: Applied to a robot, the device comprises: An acquisition unit, used to acquire multimodal data corresponding to a target scene; the target scene is a scene in which the robot is located during the process of following a target object; A determination unit, configured to determine a stain state corresponding to a slippery stain in the target scene based on the multimodal data; A first generating unit, configured to generate risk assessment information of the target object based on a movement trend of the target object and a stain state corresponding to the slippery stain; The second generating unit is configured to generate a target strategy to be executed based on a movement trend of the target object and the stain state when the risk assessment information indicates that the slippery stain causes a safety risk to the target object.

10. A robot, characterized in that: The robot comprises a processor, an image acquisition device, and an actuator, wherein: The image acquisition device is used to obtain multimodal data corresponding to a target scene; the target scene is the scene in which the robot is located during the process of following the target object; The processor is configured to determine, based on the multimodal data, a stain state corresponding to a slippery stain in the target scene; generate risk assessment information of the target object based on a motion trend of the target object and the stain state corresponding to the slippery stain; and generate a target strategy to be executed based on the motion trend of the target object and the stain state when the risk assessment information indicates that the slippery stain causes a safety risk to the target object; The executor is used to execute the target strategy.

Images