Intelligent robot control method, device, server, robot and storage medium

Abstract

The embodiment of the present invention provides an intelligent robot control method, device, server, robot and storage medium, the method comprising: receiving sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting the touch operation of the user through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot; based on the sensor data and the touch operation data, determining the multimodal reaction behavior data corresponding to the intelligent robot, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data and display data, and the display data is displayed through the flexible screen; sending the multimodal reaction behavior data to the intelligent robot, so that the intelligent robot performs reaction output based on the multimodal reaction behavior data. Enriching the external output expression form of the intelligent robot.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent robot control method, device, server, robot and storage medium. Background Art

[0002] With the development of science and technology, more and more work is being transformed from manual work to intelligent robots. There are many types of intelligent robots, and they can provide a variety of functional services. At present, a common intelligent robot is, for example, a smart speaker. When a user issues a voice command to a smart speaker, the smart speaker can recognize the user's intention and perform tasks that match the user's intention. For example, a user asks a smart speaker to play song A. After receiving the command, the smart speaker obtains the audio data of song A through the network and then plays the audio data. The interaction that can be achieved between the intelligent robot in the related technology and the user is relatively simple, and the flexibility and diversity of the interaction needs to be improved. Summary of the invention

[0003] Embodiments of the present invention provide an intelligent robot control method, device, server, robot and storage medium, which are used to improve the flexibility and diversity of the interaction between the intelligent robot and a user.

[0004] In a first aspect, an embodiment of the present invention provides an intelligent robot control method, which is applied to a server and includes:

[0005] Receiving sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting a user's touch operation through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot;

[0006] Based on the sensing data and the touch operation data, determining multimodal reaction behavior data corresponding to the intelligent robot, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen;

[0007] The multimodal reaction behavior data is sent to the intelligent robot, so that the intelligent robot performs reaction output based on the multimodal reaction behavior data.

[0008] Optionally, determining the multimodal reaction behavior data corresponding to the intelligent robot based on the sensing data and the touch operation data includes:

[0009] Preprocessing the sensor data;

[0010] The preprocessed sensing data and the touch operation data are input into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot.

[0011] Optionally, the sensor data includes image data captured by an image capturing device and / or voice reception data picked up by a voice pickup device;

[0012] The preprocessing of the sensor data includes:

[0013] performing semantic extraction on the image data; and / or,

[0014] The voice reception data is converted into text data.

[0015] Optionally, after inputting the preprocessed sensing data and the touch operation data into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot, the method further includes:

[0016] Based on preset rules, determining reference reaction behavior data and behavior constraint conditions corresponding to the preprocessed sensing data and the touch operation data;

[0017] Comparing the reference reaction behavior data with the multimodal reaction behavior data, and supplementing the multimodal reaction behavior data according to the comparison result;

[0018] The data in the multimodal reaction behavior data that do not meet the behavior constraint conditions are eliminated.

[0019] Optionally, after inputting the preprocessed sensing data and the touch operation data into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot, the method further includes:

[0020] Obtain the business logic currently being executed by the intelligent robot;

[0021] Based on the business logic, the multimodal response behavior data is adjusted.

[0022] In a second aspect, an embodiment of the present invention provides an intelligent robot control method, the method being applied to an intelligent robot, the intelligent robot comprising different types of sensors, and a flexible screen as all or part of the outer shell of the intelligent robot, the method comprising:

[0023] Collecting sensing data through the different types of sensors, and generating touch operation data through the user's touch operation detected by the flexible screen;

[0024] Uploading the sensing data and the touch operation data to a server;

[0025] Receiving multimodal reaction behavior data corresponding to the sensing data and the touch operation data sent by the server, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen;

[0026] A response output is performed based on the multimodal response behavior data.

[0027] Optionally, the sensor includes an image capturing device and / or a voice pickup device, and the sensing data includes image data captured by the image capturing device and / or voice reception data picked up by the voice pickup device;

[0028] After collecting the sensing data through the different types of sensors, the method further includes:

[0029] performing semantic extraction on the image data; and / or,

[0030] The voice reception data is converted into text data.

[0031] Optionally, the display data includes two-dimensional display data or three-dimensional display data.

[0032] Optionally, the intelligent robot includes a whole flexible screen or a plurality of flexible screens.

[0033] Optionally, when the intelligent robot includes a plurality of flexible screens, the performing reaction output based on the multimodal reaction behavior data includes:

[0034] Based on the current position of each flexible screen, determine the sub-display data in the display data corresponding to each flexible screen respectively;

[0035] Control each flexible screen to display the corresponding sub-display data.

[0036] Optionally, the method further comprises:

[0037] Detecting a division operation of the user dividing the main display area into sub-display areas through the flexible screen;

[0038] In response to the division operation, the sub-display area is divided from the main display area, and different display data are displayed in the main display area and the sub-display area respectively.

[0039] In a third aspect, an embodiment of the present invention provides an intelligent robot control device, which is applied to a server and includes:

[0040] A receiving module, used to receive sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting a user's touch operation through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot;

[0041] A determination module, configured to determine multimodal reaction behavior data corresponding to the intelligent robot based on the sensing data and the touch operation data, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen;

[0042] A sending module is used to send the multimodal reaction behavior data to the intelligent robot, so that the intelligent robot performs reaction output based on the multimodal reaction behavior data.

[0043] Optionally, the determining module is used to:

[0044] Preprocessing the sensor data;

[0045] The preprocessed sensing data and the touch operation data are input into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot.

[0046] Optionally, the sensor data includes image data captured by an image capturing device and / or voice reception data picked up by a voice pickup device;

[0047] The determining module is used to:

[0048] performing semantic extraction on the image data; and / or,

[0049] The voice reception data is converted into text data.

[0050] Optionally, the device further comprises an optimization module, wherein the optimization module is configured to:

[0051] Based on preset rules, determining reference reaction behavior data and behavior constraint conditions corresponding to the preprocessed sensing data and the touch operation data;

[0052] Comparing the reference reaction behavior data with the multimodal reaction behavior data, and supplementing the multimodal reaction behavior data according to the comparison result;

[0053] The data in the multimodal reaction behavior data that do not meet the behavior constraint conditions are eliminated.

[0054] Optionally, the optimization module is further used to:

[0055] Obtain the business logic currently being executed by the intelligent robot;

[0056] Based on the business logic, the multimodal response behavior data is adjusted.

[0057] In a fourth aspect, an embodiment of the present invention provides an intelligent robot control device, the device is applied to an intelligent robot, the intelligent robot includes different types of sensors, and a flexible screen as all or part of the outer shell of the intelligent robot, the device includes:

[0058] A collection module, used to collect sensing data through the different types of sensors, and generate touch operation data through the user's touch operation detected by the flexible screen;

[0059] A sending module, used for uploading the sensing data and the touch operation data to a server;

[0060] A receiving module, configured to receive multimodal reaction behavior data corresponding to the sensing data and the touch operation data sent by the server, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen;

[0061] An output module is used to perform reaction output based on the multimodal reaction behavior data.

[0062] Optionally, the sensor includes an image capturing device and / or a voice pickup device, and the sensing data includes image data captured by the image capturing device and / or voice reception data picked up by the voice pickup device;

[0063] The device further comprises a preprocessing module, wherein the preprocessing module is used to:

[0064] performing semantic extraction on the image data; and / or,

[0065] The voice reception data is converted into text data.

[0066] Optionally, the display data includes two-dimensional display data or three-dimensional display data.

[0067] Optionally, the intelligent robot includes a whole flexible screen or a plurality of flexible screens.

[0068] Optionally, when the intelligent robot includes a plurality of flexible screens, the output module is used to:

[0069] Based on the current position of each flexible screen, determine the sub-display data in the display data corresponding to each flexible screen respectively;

[0070] Control each flexible screen to display the corresponding sub-display data.

[0071] Optionally, the device further includes a division module, wherein the division module is configured to:

[0072] Detecting a division operation of the user dividing the main display area into sub-display areas through the flexible screen;

[0073] In response to the division operation, the sub-display area is divided from the main display area, and different display data are displayed in the main display area and the sub-display area respectively.

[0074] In a fifth aspect, an embodiment of the present invention provides a server, comprising a processor and a memory, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the intelligent robot control method in the first aspect.

[0075] In the sixth aspect, an embodiment of the present invention provides an intelligent robot, which includes a processor and a memory, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the intelligent robot control method in the second aspect.

[0076] In the seventh aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of a server, the processor can at least implement the intelligent robot control method in the first aspect.

[0077] In the eighth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an intelligent robot, the processor can at least implement the intelligent robot control method in the second aspect.

[0078] By adopting the present invention, it is possible to obtain sensor data collected by different sensors, and it is possible to detect the user's touch operation through a flexible screen and generate touch operation data. By using the sensor data and touch operation data as multimodal input data, it is possible to enrich the interaction mode with the intelligent robot, and is not limited to a single interaction mode, thereby improving the flexibility and diversity of the interaction. By processing the multimodal input data, multimodal reaction behavior data can be output accordingly, and the intelligent robot can output multimodal reaction behavior based on the multimodal reaction behavior data. For example, the intelligent robot can display the display data in the multimodal reaction behavior data through a flexible screen. In this way, the external output expression form of the intelligent robot can be enriched, and the flexibility of the intelligent robot's reaction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0080] Figure 1 A schematic diagram of a flow chart of an intelligent robot control method provided by an embodiment of the present invention;

[0081] Figure 2 A schematic diagram of an intelligent robot display provided by an embodiment of the present invention;

[0082] Figure 3 A schematic diagram showing another intelligent robot provided by an embodiment of the present invention;

[0083] Figure 4 A schematic diagram showing another intelligent robot provided by an embodiment of the present invention;

[0084] Figure 5 A schematic diagram of a flow chart of an intelligent robot control method provided by an embodiment of the present invention;

[0085] Figure 6 A schematic diagram of the structure of an intelligent robot control device provided by an embodiment of the present invention;

[0086] Figure 7 A schematic diagram of the structure of an intelligent robot control device provided by an embodiment of the present invention;

[0087] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present invention;

[0088] Fig. 9 A schematic diagram of the structure of an intelligent robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0091] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0092] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0093] The embodiment of the present invention provides an intelligent robot control method, which can be applied to the intelligent robot. Before specifically introducing the method flow, the structure of the intelligent robot is briefly introduced first, so as to facilitate understanding of the method flow provided by the embodiment of the present invention in combination with the structure of the intelligent robot.

[0094] Different types of sensors may be provided in the intelligent robot, and different types of sensors are used to collect different types of sensor data. Among them, the sensors may include but are not limited to environmental sensors, visual sensors, voice sensors, force sensors, etc. Among them, environmental sensors may include, for example, temperature sensors, humidity sensors, collision sensors, drop sensors, inertial navigation sensors, etc. Visual sensors may include, for example, at least one group of depth vision sensors, single-line or multi-line laser radars, infrared, ultrasonic, etc. Voice sensors may include, for example, microphones or microphone arrays that receive voice data. Force sensors may include, for example, torque sensors on each joint of the intelligent robot. Through these sensors, the intelligent robot can perceive the surrounding environment.

[0095] The outer shell of the intelligent robot can be set as a flexible screen, and the flexible screen can serve as all or part of the outer shell of the intelligent robot. Based on this, the flexible screen can fully or partially surround the internal components of the intelligent robot.

[0096] It is understandable that the flexible screen can be bent so that the flexible screen can form a certain shape to serve as the outer shell of the intelligent robot. In some optional embodiments, the humanoid intelligent robot can be composed of the following components: head (e.g., the entire face, the back of the head, the side of the head, etc., wherein the face can include eyes), neck joints, arms (e.g., upper arms, lower arms, elbow joints, etc.), humanoid hands (e.g., 2-5 fingers, 2-3 joints in each finger), multi-degree-of-freedom waist, single leg, double legs, knee joints, feet, wheeled chassis of various shapes, etc. The outer shells of the components of the intelligent robot listed above can all be set as flexible screens, and some of the components can be selected from them to set their outer shells as flexible screens, or the outer shells of all the components can be set as flexible screens.

[0097] For the sake of convenience of maintenance and other factors, some sensors of the intelligent robot will be set in a position that is easily exposed to the outside, and the sensors will be covered inside the intelligent robot by a cover. Among them, the cover can be set in a position aligned with the shell, or in a space recessed in the shell. In this way, after opening the cover, the sensor can be exposed to the outside. In some optional embodiments, since the flexible screen replaces the original shell, in order to allow the flexible screen to be covered on the outer surface of the intelligent robot as completely as possible, it can be set to cover the sensors originally installed on the outside. In this way, it is possible to avoid hollowing out a small area of ​​the flexible screen and maintain the integrity of the flexible screen. Among them, the sensors originally installed on the outside may include microphones (arrays), various visual cameras, obstacle avoidance cameras, laser radars, environmental sensors, inertial navigation, torque sensors, collision sensors, drop sensors, etc.

[0098] The flexible screen provided in the intelligent robot can be used as an input component or an output component. When it is used as an input component, the flexible screen can detect the user's touch operation. When it is used as an output component, the flexible screen can display display data. It is worth noting that the flexible screen can replace the tactile sensor to detect the user's touch operation. Compared with the tactile sensor, if the intelligent robot is required to sense the user's touch from multiple directions, it is necessary to set more tactile sensors in multiple directions. However, due to space limitations, cost limitations, etc., more tactile sensors will not be installed on the intelligent robot, which results in the user only being able to touch certain structural parts of the intelligent robot. If the user touches the structural part of the intelligent robot that is not provided with a tactile sensor, the intelligent robot cannot sense the user's touch. With the present invention, a flexible screen can be laid on the surface of the intelligent robot over a large area. By replacing the original tactile sensor with the flexible screen, the flexible screen, as a component that can detect the user's touch operation, can well solve the problem of tactile sensors throughout the intelligent robot. As long as the user touches the structural part of the intelligent robot provided with a flexible screen, the intelligent robot can sense such touch.

[0099] At the same time, the flexible screen can receive the user's touch operation more flexibly, unlike the tactile sensor which can only receive a relatively single touch operation. The touch operation that the flexible screen can detect may include single-point touch, multi-point touch, single-point touch with a certain distance sliding, multi-point touch with a certain distance sliding, etc.

[0100] Figure 1 A flow chart of an intelligent robot control method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0101] 101. Collect sensing data through different types of sensors, and generate touch operation data through the user's touch operation detected by the flexible screen.

[0102] 102. Upload the sensing data and touch operation data to the server.

[0103] 103. Receive multimodal reaction behavior data corresponding to the sensor data and the touch operation data sent by the server, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through a flexible screen.

[0104] 104. Output response based on multimodal response behavior data.

[0105] In practical applications, different types of sensor data can be collected through different types of sensors. For example, when the sensor is a voice pickup device (microphone, etc.), voice reception data can be collected. When the sensor is an image capture device (at least one group of depth vision sensors, etc.), image data can be captured. At the same time, the user's touch operation can be detected by the flexible screen, and touch operation data can be generated based on the touch operation. Then, the sensor data and touch operation data can be used as multimodal input data. It can be understood that the "multimodality" here can be interpreted as different dimensions and different types.

[0106] It should be noted that in actual applications, not all sensors can collect corresponding sensor data at the same time. If a sensor does not collect valid sensor data, the sensor data that can be collected by the sensor may not be used as multimodal input data. For example, if a user does not speak to the intelligent robot but only waves to the intelligent robot, the voice pickup device in the intelligent robot may not collect valid voice reception data. At this time, the voice reception data is not used as one of the multimodal input data. In general, the dimensions of data included in the multimodal input data are determined by the specific usage in the actual application scenario.

[0107] After the intelligent robot collects sensor data and touch operation data through the sensors installed on itself and the flexible screen, it can upload these data to the server for processing. In the actual application scenario, assuming that a user inputs multimodal input data to the intelligent robot through his own behavior, the intelligent robot needs to make a corresponding response based on the user's behavior. What kind of specific response the intelligent robot needs to make is based on the analysis results of the user's behavior. This analysis logic can be set in the server, and the server gives the response that the intelligent robot should output to the user. The response that the intelligent robot should output to the user can be represented by multimodal behavior response data. The process of how the server outputs multimodal behavior response data based on multimodal input data will be described in detail in the next embodiment, and will not be described in detail here.

[0108] The intelligent robot can receive multimodal behavior response data sent by the server, and perform response output based on the multimodal response behavior data. Among them, the multimodal behavior response data can be composed of one or more of action data, voice output data, and display data. Based on this, the response that the intelligent robot can output can be diversified and is not limited to the response results of a single dimension. Based on this, the use of the present invention enriches the form of the intelligent robot's response output and improves the diversification of the intelligent robot's response output. Among them, the display data can be displayed through a flexible screen.

[0109] For example, suppose that during a conversation between a user and an intelligent robot, the intelligent robot wants to express to the user that it is feeling very excited. At this time, the intelligent robot can broadcast the voice output data "I feel very excited" through voice broadcasting, and simultaneously, the intelligent robot can also play a video of the rolling sea waves in the display data on the flexible screen to vividly convey to the user what the intelligent robot wants to express.

[0110] Optionally, the display data may include two-dimensional display data or three-dimensional display data. If the three-dimensional display data is displayed through a flexible screen, the user can see a three-dimensional image from every angle of the intelligent robot.

[0111] In some optional embodiments, the intelligent robot may be controlled to broadcast voice and display display data synchronously or asynchronously, wherein the broadcast voice output data may be related or unrelated to the specific content of the display data.

[0112] In some optional embodiments, the intelligent robot can also be controlled to perform some actions in a time-series manner while displaying display data through a flexible screen. It can also be considered that the actions performed by the intelligent robot are synchronized with the displayed display data. Alternatively, the intelligent robot can also be controlled to perform some actions in a time-series manner while broadcasting voice output data while displaying display data through a flexible screen. The display data may be related to the broadcast voice output data and the executed action data or not.

[0113] After collecting the sensor data, the intelligent robot can directly upload the sensor data to the server for processing, or pre-process the sensor data locally and then upload the pre-processed sensor data to the server. Therefore, the pre-processing process can also be completed in the server. Optionally, the sensor provided in the intelligent robot may include an image capture device and / or a voice pickup device, and the sensor data includes image data captured by the image capture device and / or voice reception data picked up by the voice pickup device. Accordingly, after collecting sensor data through different types of sensors, the intelligent robot can also perform semantic extraction on the image data; and / or, convert the voice reception data into text data.

[0114] It should be noted that, since the preprocessing process can also be executed in the server, the preprocessing process will be introduced in the method flow executed in the server later, and will not be expanded here.

[0115] Optionally, the flexible screen provided in the intelligent robot can be a whole flexible screen or a plurality of spliced ​​flexible screens. For example, a larger area of ​​the chest of the intelligent robot can be set as a whole flexible screen. Alternatively, each component of the intelligent robot can be set as a flexible screen, for example, the left and right arms, chest, and legs can be set as flexible screens, and they can be spliced ​​together for use. Of course, when the intelligent robot is provided with multiple flexible screens, these flexible screens can be spliced ​​together for use as a whole or can be used separately. The images displayed on different flexible screens can be the same or different, and some of the multiple flexible screens can be set not to display any images. In addition, it should be noted that for a flexible screen, the image displayed therein can be stretched, and the stretching operation method includes but is not limited to unidirectional, bidirectional, and multi-directional stretching.

[0116] When multiple flexible screens are spliced ​​together for display, optionally, when the intelligent robot includes multiple flexible screens, the process of reaction output based on multimodal reaction behavior data can be implemented as follows: based on the current position of each flexible screen, determine the sub-display data in the display data corresponding to each flexible screen; control each flexible screen to display its corresponding sub-display data.

[0117] by Figure 2 The example shown is used to introduce the process of splicing multiple flexible screens together for display, such as Figure 2 As shown, the left and right arms, chest, and legs of the intelligent robot are respectively set as flexible screens. When the arms of the intelligent robot are drooping and standing upright, the flexible screens respectively set at the corresponding positions of the left and right arms, chest, and legs can be spliced ​​into a polygon A. The circumscribed rectangle A' or the bounding box of polygon A can be determined, and the display data to be displayed is filled in the circumscribed rectangle A'. Since some areas in the circumscribed rectangle A' are covered with flexible screens, and some areas are not covered with flexible screens, the corresponding sub-display data in the display data is normally displayed for the areas covered with the flexible screen, and the corresponding sub-display data in the display data may not be displayed for the areas not covered with the flexible screen. The flexible screens set at the corresponding positions of the left and right arms, chest, and legs of the intelligent robot can each display the corresponding sub-display data in the display data.

[0118] Since the intelligent robot may also make some actions while displaying, when the intelligent robot makes actions, the positions of different flexible screens will shift with the set components. After the flexible screen shifts, the above operation can be repeated to re-determine the sub-display data in the display data corresponding to each flexible screen based on the position of each flexible screen, and control each flexible screen to display its corresponding sub-display data.

[0119] For example, if Figure 3As shown, the intelligent robot changes from a state with its arms hanging down and standing vertically to a state with its arms opened horizontally to both sides, so that the flexible screens respectively arranged at the positions corresponding to the left and right arms, chest, and legs form a "T" shape. At this time, the flexible screens respectively arranged at the positions corresponding to the left and right arms, chest, and legs can be spliced ​​into a polygon B. The circumscribed rectangle B' or bounding box of polygon B can be determined, and the display data to be displayed will fill the circumscribed rectangle B'. Since the area of ​​the circumscribed rectangle B' is smaller than the circumscribed rectangle A', in order for the display data to still fill the circumscribed rectangle B', it is necessary to stretch the display data in the circumscribed rectangle A' so that the stretched display data can fill the circumscribed rectangle B'.

[0120] Since some areas in the circumscribed rectangle B' are covered with the flexible screen and some areas are not covered with the flexible screen, the corresponding sub-display data in the display data is normally displayed for the areas covered with the flexible screen, and the corresponding sub-display data in the display data may not be displayed for the areas not covered with the flexible screen.

[0121] It can be seen that in this state of the intelligent robot, there is a large area in the circumscribed rectangle B' that is not covered by the flexible screen, and accordingly some information in the display data cannot be displayed. If the undisplayed information is large and important, the intelligent robot can be controlled to move appropriately so that the flexible screen can cover most or all of the area in the circumscribed rectangle B' in a time-sharing manner to display more information. For example, Figure 4 As shown, the arms of the intelligent robot can be moved up and down, so that the area swept by the arms can be enclosed in a fan shape. When the arms sweep over a certain position, the sub-display data corresponding to the position can be displayed, so that the effect of time-sharing display can be achieved. When the arms of the intelligent robot move fast enough, it is difficult for the human eye to distinguish the position of the arms. At this time, from a visual effect, the user can see all the contents in the fan-shaped area at the same time, thereby achieving a dynamic display effect.

[0122] The flexible screen provided in the embodiment of the present invention can also support a "picture-in-picture" display effect. Optionally, a division operation of a user dividing a main display area into a sub-display area through the flexible screen is detected; in response to the division operation, a sub-display area is divided in the main display area, and different display data is displayed in the main display area and the sub-display area respectively.

[0123] In actual applications, the flexible screen can be divided into a main display area and a sub-display area to achieve "picture-in-picture". The position, size, shape, etc. of the sub-display area can be specified by the user through touch operations, wherein the touch operations for setting the sub-display area may include but are not limited to multi-points, multi-lines, and drawing irregular graphics. Among them, the multi-point operation mode can be understood as touching multiple points in the flexible screen and uniquely determining a polygon through the positions of the multiple points. For example, the user can touch two points in the flexible screen, and the two points can be used as two points on the diagonal of a rectangle, so a rectangle can be uniquely determined based on the two points. The multi-line operation mode can be understood as drawing multiple lines in the flexible screen, and the multiple lines can form a polygon. Drawing irregular graphics can be understood as the user can freely draw closed areas of any shape.

[0124] In addition, it should be noted that "picture-in-picture" can be set in a single flexible screen, or "picture-in-picture" can be set after multiple flexible screens are spliced ​​together.

[0125] Another embodiment of the present invention provides an intelligent robot control method, which can be applied to a server. Figure 5 A flow chart of an intelligent robot control method provided by an embodiment of the present invention, such as Figure 5 As shown, the method comprises the following steps:

[0126] 501. Receive sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting the user's touch operation through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot.

[0127] 502. Based on the sensing data and the touch operation data, determine the multimodal reaction behavior data corresponding to the intelligent robot, wherein the multimodal reaction behavior data includes at least any one of the action data, the voice output data, and the display data, and the display data is displayed through the flexible screen.

[0128] 503. Send the multimodal reaction behavior data to the intelligent robot, so that the intelligent robot performs reaction output based on the multimodal reaction behavior data.

[0129] The intelligent robot can collect sensor data through different types of sensors, and detect the user's touch operation through the flexible screen to generate touch operation data. The intelligent robot can upload the sensor data and touch operation data to the server, and the server can receive these data. The specific process of the intelligent robot collecting sensor data and obtaining touch operation data has been introduced in detail in the previous embodiment, and will not be repeated here.

[0130] The server can output multimodal reaction behavior data accordingly based on the sensor data and touch operation data uploaded by the intelligent robot. Optionally, the process of determining the multimodal reaction behavior data can be implemented as follows: obtaining preprocessed sensor data; inputting the preprocessed sensor data and touch operation data into a pre-trained neural network model to obtain the multimodal reaction behavior data corresponding to the intelligent robot.

[0131] The sensor data may be preprocessed in the intelligent robot, or may be preprocessed locally on the server, which is not limited in the embodiments of the present invention.

[0132] Optionally, the sensor data may include image data captured by an image capturing device and / or voice receiving data captured by a voice capturing device. Accordingly, the process of preprocessing the sensor data may be implemented as: performing semantic extraction on the image data; and / or converting the voice receiving data into text data.

[0133] Among them, the image data contains a lot of information, some of which is helpful for understanding the environment around the intelligent robot, while some information interferes with the process of understanding the environment around the intelligent robot. Based on this, semantic extraction can be performed on the image data to extract the important information of interest in the image data. On the other hand, for the neural network model, it can more effectively utilize the semantic extraction results of the image data. If the original image data is directly input into the neural network model, the neural network model may not be able to efficiently extract the useful features of the image data. Similar to the image data, for the neural network model, it can more effectively utilize the text data converted based on the voice reception data. Based on this, before the image data and the voice reception data are input into the neural network model, these data can be preprocessed so that the neural network model can more efficiently utilize the input information.

[0134] After preprocessing the sensor data, it can be considered that a sequence of high-dimensional vectors can be obtained. Then, the sequence of high-dimensional vectors can be input into the neural network model, and the neural network model can output the multimodal response behavior data required for the intelligent robot to respond to the environment.

[0135] Among them, the neural network model can be constructed using the following principles and methods: simple operation fusion method, fusion method based on attention mechanism, fusion method based on bilinear pooling, etc.

[0136] The output of the neural network model may be an identifier of the display data. A multimedia resource library may also be provided in the server, and corresponding display data may be searched in the multimedia resource library based on the identifier of the display data, and the display data may be sent to the intelligent robot. The multimedia resource library may store different types of data such as voice, text, image, audio and video.

[0137] For example, if you want the intelligent robot to express the scene of the surging emotions in its heart, you can play a video of the rolling sea waves on the flexible screen. First, the neural network model can output the logo of the video of the rolling sea waves, and then based on the logo, it can find the video of the rolling sea waves from the multimedia resource library and send the video of the rolling sea waves to the intelligent robot.

[0138] Optionally, in order to enable the intelligent robot to better respond to the environment, after obtaining the multimodal reaction behavior data corresponding to the intelligent robot, it is also possible to determine the reference reaction behavior data and behavior constraints corresponding to the preprocessed sensor data and touch operation data based on preset rules; compare the reference reaction behavior data and the multimodal reaction behavior data, and supplement the multimodal reaction behavior data according to the comparison results; and eliminate the data in the multimodal reaction behavior data that does not meet the behavior constraints.

[0139] In practical applications, reference reaction behavior data and behavior constraints corresponding to the preprocessed sensor data and touch operation data can be determined based on the knowledge base, set rules, etc. The reference reaction behavior data can be used as a supplement to the multimodal reaction behavior data, and the behavior constraints can be used as behavior constraints.

[0140] For example, after seeing a child, the intelligent robot can say hello to him. Generally speaking, if the intelligent robot is facing an adult, it can say hello to him normally. If it is facing a child, it can bend down before saying hello, so that it can keep the same eye level with the child as much as possible, which makes it more friendly. The bending down action is the supplementary data of the multimodal reaction behavior data in the reference reaction behavior data.

[0141] For example, after the intelligent robot greets an adult, the business logic it will execute next may be to recommend a business product to the adult, such as a financial product. However, if the intelligent robot is facing a child, it may not be able to execute the above business logic next, and it may instead execute other logic, such as asking the child what cartoons he wants to watch. Not recommending business products to children is a constraint on the multimodal response behavior data.

[0142] Optionally, in order to further enable the intelligent robot to better respond to the environment, after obtaining the multimodal reaction behavior data corresponding to the intelligent robot, the business logic currently being executed by the intelligent robot can also be obtained; based on the business logic, the multimodal reaction behavior data can be adjusted.

[0143] For example, suppose the intelligent robot is currently executing the business logic of pouring coffee for user A, and the priority of this business logic is high. In the process of executing the business logic, suppose the intelligent robot encounters user B on the way to the workbench where the coffee is poured. At this time, the intelligent robot can temporarily suspend the task of pouring coffee for user A. The intelligent robot can greet user B first, and then continue to perform the task of pouring coffee for user A. If the intelligent robot is not currently executing the main business logic, for example, it is not currently performing the task of pouring coffee for user A, when the intelligent robot encounters user B, in addition to greeting user B, it can also ask user B if he needs any help, etc. It can be seen that the business logic currently being executed by the intelligent robot can also affect the final response behavior.

[0144] In addition to adjusting the multimodal response behavior data according to the business logic, the multimodal response behavior data can also be adjusted according to the control instructions given by the back-end personnel. It is worth noting that the knowledge base, the set rules, the business logic, and the control instructions given by the back-end personnel can form a nested control logic to ultimately adjust the multimodal response behavior data.

[0145] By adopting the present invention, it is possible to obtain sensor data collected by different sensors, and it is possible to detect the user's touch operation through a flexible screen and generate touch operation data. By using the sensor data and touch operation data as multimodal input data, it is possible to enrich the interaction mode with the intelligent robot, and is not limited to a single interaction mode, thereby improving the flexibility and diversity of the interaction. By processing the multimodal input data, multimodal reaction behavior data can be output accordingly, and the intelligent robot can output multimodal reaction behavior based on the multimodal reaction behavior data. For example, the intelligent robot can display the display data in the multimodal reaction behavior data through a flexible screen. In this way, the external output expression form of the intelligent robot can be enriched, and the flexibility of the intelligent robot's reaction can be improved.

[0146] The following will describe in detail the intelligent robot control device of one or more embodiments of the present invention. Those skilled in the art will appreciate that these intelligent robot control devices can be configured using commercially available hardware components through the steps taught in this solution.

[0147] Figure 6Schematic diagram of the structure of an intelligent robot control device provided by an embodiment of the present invention, the device is applied to a server, such as Figure 6 As shown, the device comprises:

[0148] A receiving module 61 is used to receive sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting a user's touch operation through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot;

[0149] A determination module 62, configured to determine multimodal reaction behavior data corresponding to the intelligent robot based on the sensing data and the touch operation data, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen;

[0150] The sending module 63 is used to send the multimodal reaction behavior data to the intelligent robot, so that the intelligent robot performs reaction output based on the multimodal reaction behavior data.

[0151] Optionally, the determining module 62 is used to:

[0152] Preprocessing the sensor data;

[0153] The preprocessed sensing data and the touch operation data are input into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot.

[0154] Optionally, the sensor data includes image data captured by an image capturing device and / or voice reception data picked up by a voice pickup device;

[0155] The determining module 62 is used to:

[0156] performing semantic extraction on the image data; and / or,

[0157] The voice reception data is converted into text data.

[0158] Optionally, the device further comprises an optimization module, wherein the optimization module is configured to:

[0159] Based on preset rules, determining reference reaction behavior data and behavior constraint conditions corresponding to the preprocessed sensing data and the touch operation data;

[0160] Comparing the reference reaction behavior data with the multimodal reaction behavior data, and supplementing the multimodal reaction behavior data according to the comparison result;

[0161] The data in the multimodal reaction behavior data that do not meet the behavior constraint conditions are eliminated.

[0162] Optionally, the optimization module is further used to:

[0163] Obtain the business logic currently being executed by the intelligent robot;

[0164] Based on the business logic, the multimodal response behavior data is adjusted.

[0165] Figure 7 The present invention provides a schematic diagram of the structure of an intelligent robot control device, which is applied to an intelligent robot. The intelligent robot includes different types of sensors and a flexible screen as all or part of the outer shell of the intelligent robot. Figure 7 As shown, the device comprises:

[0166] A collection module 71, configured to collect sensing data through the different types of sensors, and generate touch operation data through the user's touch operation detected by the flexible screen;

[0167] A sending module 72, used for uploading the sensing data and the touch operation data to a server;

[0168] A receiving module 73, configured to receive multimodal reaction behavior data corresponding to the sensing data and the touch operation data sent by the server, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen;

[0169] The output module 74 is used to perform reaction output based on the multimodal reaction behavior data.

[0170] Optionally, the sensor includes an image capturing device and / or a voice pickup device, and the sensing data includes image data captured by the image capturing device and / or voice reception data picked up by the voice pickup device;

[0171] The device further comprises a preprocessing module, wherein the preprocessing module is used to:

[0172] performing semantic extraction on the image data; and / or,

[0173] The voice reception data is converted into text data.

[0174] Optionally, the display data includes two-dimensional display data or three-dimensional display data.

[0175] Optionally, the intelligent robot includes a whole flexible screen or a plurality of flexible screens.

[0176] Optionally, when the intelligent robot includes a plurality of flexible screens, the output module 74 is used to:

[0177] Based on the current position of each flexible screen, determine the sub-display data in the display data corresponding to each flexible screen respectively;

[0178] Control each flexible screen to display the corresponding sub-display data.

[0179] Optionally, the device further includes a division module, wherein the division module is configured to:

[0180] Detecting a division operation of the user dividing the main display area into sub-display areas through the flexible screen;

[0181] In response to the division operation, the sub-display area is divided from the main display area, and different display data are displayed in the main display area and the sub-display area respectively.

[0182] Figure 6 , 7 The device shown can perform the aforementioned Figure 5 , Figures 1 to 4 The intelligent robot control method provided in the illustrated embodiment, the detailed execution process and technical effects can be found in the description of the aforementioned embodiment, which will not be repeated here.

[0183] In one possible design, the above Figure 6 The structure of the intelligent robot control device shown can be implemented as a server, such as Figure 8 As shown, the server may include: a processor 91 and a memory 92. The memory 92 stores executable code, and when the executable code is executed by the processor 91, the processor 91 can at least implement the above-mentioned Figure 5 The intelligent robot control method provided in the illustrated embodiment.

[0184] Optionally, the server may further include a communication interface 93 for communicating with other devices.

[0185] In one possible design, the above Figure 7 The structure of the intelligent robot control device shown can be realized as an intelligent robot, such as Fig. 9As shown, the intelligent robot may include: a processor 91' and a memory 92'. The memory 92' stores executable code. When the executable code is executed by the processor 91', the processor 91' can at least implement the above-mentioned Figures 1 to 4 The intelligent robot control method provided in the illustrated embodiment.

[0186] Optionally, the intelligent robot may further include a communication interface 93' for communicating with other devices.

[0187] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an intelligent robot, the processor can at least implement the above-mentioned Figures 1 to 4 The intelligent robot control method provided in the illustrated embodiment.

[0188] An embodiment of the present invention provides a non-transitory machine-readable storage medium, wherein an executable code is stored on the non-transitory machine-readable storage medium. When the executable code is executed by a processor of a server, the processor can at least implement the above-mentioned Figure 5 The intelligent robot control method provided in the illustrated embodiment.

[0189] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0190] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0191] The intelligent robot control method provided in the embodiment of the present invention can be executed by a certain program / software, and the program / software can be provided by the network side. The server mentioned in the above embodiment can download the program / software to a local non-volatile storage medium, and when it needs to execute the above intelligent robot control method, the program / software is read into the memory by the CPU, and then the program / software is executed by the CPU to implement the intelligent robot control method provided in the above embodiment. The execution process can refer to the above Figures 1 to 4 Instructions in .

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent robot control method, characterized in that: Applied to a server, the method comprises: Receiving sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting a user's touch operation through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot; Based on the sensing data and the touch operation data, determining multimodal reaction behavior data corresponding to the intelligent robot, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen; The multimodal reaction behavior data is sent to the intelligent robot so that the intelligent robot performs reaction output based on the multimodal reaction behavior data; wherein, when the intelligent robot includes multiple flexible screens, based on the current position of each flexible screen, the sub-display data in the display data corresponding to each flexible screen is determined; each flexible screen is controlled to display its corresponding sub-display data; when the flexible screen is shifted, the sub-display data in the display data corresponding to each flexible screen is determined again based on the position of each flexible screen, and each flexible screen is controlled to display its corresponding sub-display data; for the circumscribed rectangle surrounding the multiple flexible screens, the area covered by the flexible screen in the circumscribed rectangle displays the corresponding sub-display data in the display data, and the area not covered by the flexible screen in the circumscribed rectangle does not display the corresponding sub-display data in the display data; if there is a large area in the circumscribed rectangle that is not covered by the flexible screen, the intelligent robot is controlled to move so that the flexible screen covers the entire area in the circumscribed rectangle in time-sharing, thereby achieving a time-sharing display effect.

2. The method according to claim 1, characterized in that The determining, based on the sensing data and the touch operation data, the multimodal reaction behavior data corresponding to the intelligent robot comprises: Preprocessing the sensor data; The preprocessed sensing data and the touch operation data are input into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot.

3. The method according to claim 2, characterized in that The sensor data includes image data captured by an image capturing device and / or voice reception data picked up by a voice pickup device; The preprocessing of the sensor data includes: Performing semantic extraction on the image data; and / or, The voice reception data is converted into text data.

4. The method according to claim 2, characterized in that: After inputting the preprocessed sensing data and the touch operation data into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot, the method further includes: Based on preset rules, determining reference reaction behavior data and behavior constraint conditions corresponding to the preprocessed sensing data and the touch operation data; Comparing the reference reaction behavior data with the multimodal reaction behavior data, and supplementing the multimodal reaction behavior data according to the comparison result; The data in the multimodal reaction behavior data that do not meet the behavior constraint conditions are eliminated.

5. The method according to claim 2, characterized in that: After inputting the preprocessed sensing data and the touch operation data into a pre-trained neural network model to obtain multimodal reaction behavior data corresponding to the intelligent robot, the method further includes: Obtain the business logic currently being executed by the intelligent robot; Based on the business logic, the multimodal response behavior data is adjusted.

6. An intelligent robot control method, characterized in that: Applied to an intelligent robot, the intelligent robot includes different types of sensors and a flexible screen as all or part of the outer shell of the intelligent robot, the method includes: Collecting sensing data through the different types of sensors, and generating touch operation data through the user's touch operation detected by the flexible screen; Uploading the sensing data and the touch operation data to a server; Receiving multimodal reaction behavior data corresponding to the sensing data and the touch operation data sent by the server, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen; Reaction output is performed based on the multimodal reaction behavior data; wherein, when the intelligent robot includes multiple flexible screens, based on the current position of each flexible screen, the sub-display data in the display data corresponding to each flexible screen is determined; each flexible screen is controlled to display its corresponding sub-display data; when the flexible screen is shifted, the sub-display data in the display data corresponding to each flexible screen is determined again based on the position of each flexible screen, and each flexible screen is controlled to display its corresponding sub-display data; for the circumscribed rectangle enclosing the multiple flexible screens, the area covered by the flexible screen in the circumscribed rectangle displays the corresponding sub-display data in the display data, and the area not covered by the flexible screen in the circumscribed rectangle does not display the corresponding sub-display data in the display data; if there is a large area in the circumscribed rectangle that is not covered by the flexible screen, the intelligent robot is controlled to move so that the flexible screen covers the entire area in the circumscribed rectangle in a time-sharing manner, thereby achieving a time-sharing display effect.

7. The method according to claim 6, characterized in that The sensor includes an image capturing device and / or a voice pickup device, and the sensing data includes image data captured by the image capturing device and / or voice reception data picked up by the voice pickup device; After collecting the sensing data through the different types of sensors, the method further includes: Performing semantic extraction on the image data; and / or, The voice reception data is converted into text data.

8. The method according to claim 6, characterized in that The display data includes two-dimensional display data or three-dimensional display data.

9. The method according to claim 6, characterized in that The method further comprises: Detecting a division operation of the user dividing the main display area into sub-display areas through the flexible screen; In response to the division operation, the sub-display area is divided from the main display area, and different display data are displayed in the main display area and the sub-display area respectively.

10. An intelligent robot control device, characterized in that: Applied to a server, the device comprises: A receiving module, used to receive sensor data and touch operation data uploaded by the intelligent robot, wherein the sensor data is collected by different types of sensors provided in the intelligent robot, and the touch operation data is generated by detecting a user's touch operation through a flexible screen provided in the intelligent robot, and the flexible screen forms all or part of the outer shell of the intelligent robot; A determination module, configured to determine multimodal reaction behavior data corresponding to the intelligent robot based on the sensing data and the touch operation data, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen; A sending module, used for sending the multimodal reaction behavior data to the intelligent robot, so that the intelligent robot performs reaction output based on the multimodal reaction behavior data; wherein, when the intelligent robot includes multiple flexible screens, based on the current position of each flexible screen, the sub-display data in the display data corresponding to each flexible screen is determined; each flexible screen is controlled to display its corresponding sub-display data; when the flexible screen is shifted, the sub-display data in the display data corresponding to each flexible screen is determined again based on the position of each flexible screen, and each flexible screen is controlled to display its corresponding sub-display data; for the circumscribed rectangle surrounding the multiple flexible screens, the area covered by the flexible screen in the circumscribed rectangle displays the corresponding sub-display data in the display data, and the area not covered by the flexible screen in the circumscribed rectangle does not display the corresponding sub-display data in the display data; if there is a large area in the circumscribed rectangle that is not covered by the flexible screen, the intelligent robot is controlled to move so that the flexible screen covers the entire area in the circumscribed rectangle in time-sharing, thereby achieving a time-sharing display effect.

11. An intelligent robot control device, characterized in that: Applied to an intelligent robot, the intelligent robot includes different types of sensors and a flexible screen as all or part of the outer shell of the intelligent robot, the device includes: A collection module, used to collect sensing data through the different types of sensors, and generate touch operation data through the user's touch operation detected by the flexible screen; A sending module, used for uploading the sensing data and the touch operation data to a server; A receiving module, configured to receive multimodal reaction behavior data corresponding to the sensing data and the touch operation data sent by the server, wherein the multimodal reaction behavior data at least includes any one of action data, voice output data, and display data, and the display data is displayed through the flexible screen; An output module is used to perform reaction output based on the multimodal reaction behavior data; wherein, when the intelligent robot includes multiple flexible screens, based on the current position of each flexible screen, the sub-display data in the display data corresponding to each flexible screen is determined; each flexible screen is controlled to display its corresponding sub-display data; when the flexible screen is shifted, the sub-display data in the display data corresponding to each flexible screen is determined again based on the position of each flexible screen, and each flexible screen is controlled to display its corresponding sub-display data; for the circumscribed rectangle enclosing the multiple flexible screens, the area covered by the flexible screen in the circumscribed rectangle displays the corresponding sub-display data in the display data, and the area not covered by the flexible screen in the circumscribed rectangle does not display the corresponding sub-display data in the display data; if there is a large area in the circumscribed rectangle that is not covered by the flexible screen, the intelligent robot is controlled to move so that the flexible screen covers the entire area in the circumscribed rectangle in a time-sharing manner, thereby achieving a time-sharing display effect.

12. A server, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the intelligent robot control method as described in any one of claims 1 to 5.

13. An intelligent robot, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the intelligent robot control method as described in any one of claims 6 to 9.

14. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of a server, the processor executes the intelligent robot control method as described in any one of claims 1-5.

15. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of the intelligent robot, the processor executes the intelligent robot control method as described in any one of claims 6-9.

Images