Eye structure of bionic service type humanoid robot

Abstract

The utility model discloses a bionic service type humanoid robot eye structure which comprises a sensory system and a visual system connected with the sensory system, the visual system comprises an eyebrow structure, an eyelid structure and an eyeball, and the eyebrow structure, the eyelid structure and the eyeball receive control signals sent by the sensory system to execute corresponding motions. According to the utility model, the sensing center, the sensor network, the signal processing unit and the execution unit form a sensory system, so that timely and effective feedback can be made to external environment changes, and the capability of interaction with the outside is greatly improved; meanwhile, according to the taste sensor, the humanoid robot can test and evaluate the food under the condition that no human testee exists, and the taste sensing system can simulate the function of a human taste sensing system, so that the taste of the food is automatically evaluated.

Description

Technical Field

[0001] The utility model belongs to the field of robot bionics and relates to the eye structure of a bionic service-type humanoid robot. Background Art

[0002] A "humanoid robot" can be defined as a robot that has certain attributes of a human's appearance and functionality (e.g., torso, head, arms, legs), the ability to communicate verbally with humans using speech recognition and voice synthesis, etc. This type of robot aims to reduce the cognitive distance between humans and machines.

[0003] Existing "humanoid robots" have limited ability to interact with the outside world and are unable to provide timely feedback based on changes in the external environment. Summary of the Invention

[0004] In order to overcome at least one deficiency of the prior art, the utility model provides an eye structure of a bionic service-type humanoid robot.

[0005] In order to achieve the above-mentioned purpose, the utility model adopts the following technical solutions: the eye structure of a bionic service humanoid robot is installed on the head structure, including a sensory system and a visual system connected to the sensory system, the visual system includes an eyebrow structure, an eyelid structure and an eyeball, and the eyebrow structure, eyelid structure and eyeball receive control signals sent by the sensory system to perform corresponding movements.

[0006] Furthermore, the eyebrow structure includes an eyebrow movable support and two groups of eyebrow movable components, and the two groups of eyebrow movable components are respectively connected to the eyebrow movable support.

[0007] Furthermore, the eyelid structure includes an upper eyelid support, an upper eyelid, a lower eyelid, a lower eyelid support and an eyelid movable component, and the opening and closing movements of the upper eyelid support and the lower eyelid support are controlled by the eyelid movable component.

[0008] Furthermore, the eyeball includes a body assembly and an eye movable component. The body assembly is movably installed in the middle of the eyelid structure, and the up and down movement and rotation of the body assembly are controlled by the eye movable component.

[0009] Furthermore, the eyebrow movable component includes a servo servo and an eyebrow connecting rod, the upper end of the eyebrow connecting rod is connected to an upper movable joint, and the lower end of the eyebrow connecting rod is connected to a lower movable joint. The servo servo is installed on the head structure, the upper movable joint is connected to the servo servo through an eyebrow movable pin, and the eyebrow movable bracket is provided with an eyebrow fixing pin, and the lower movable joint is connected to the eyebrow fixing pin.

[0010] Furthermore, the upper eyelid is fixed on the upper eyelid bracket, and the lower eyelid is fixed on the lower eyelid bracket. The upper eyelid bracket and the lower eyelid bracket are respectively provided with tooth ends at the left and right ends, and the two tooth ends are coaxial and meshingly arranged inside and outside.

[0011] Furthermore, the eyelid movable component includes an eyelid opening and closing rotating shaft and a servo servo, the servo servo is installed on the head structure, the output shaft of the servo servo is fixedly connected to the eyelid opening and closing rotating shaft, and the eyelid opening and closing rotating shaft is fixedly connected to the tooth end located on the inner side.

[0012] Furthermore, the main body assembly includes an eyeball-shaped shell, an eyeball-shaped shell back cover and a back cover wire collection slot locking nut. A visual sensor, a variable-focus movable motor seat, an integrated circuit board and a rubber ring for fixing the connecting rod seat are fixed inside the eyeball-shaped shell in sequence. The eyeball-shaped shell back cover is fixed to the end of the eyeball-shaped shell and is locked by the back cover wire collection slot locking nut.

[0013] Furthermore, the eye movement component includes an eyeball support rod, an eyeball connecting rod and a servo servo. The eyeball support rod is located in the middle to connect the eyeball and the eyeball connecting block. The eyeball connecting block is installed on the skull structure. The servo servo is installed on the eyeball connecting block. Active joints are provided at both ends of the eyeball connecting rod. The active joint at one end is connected to the eyeball contour shell through a fixed pin, and the other end is connected to the servo servo through a active pin. A group of servo servos controls the movement of a group of eyeball connecting rods.

[0014] Furthermore, the visual system also includes an eye shell, and the eyebrow structure, eyelid structure and eyeball are installed inside the eye shell.

[0015] In summary, the benefits of the present invention are:

[0016] The utility model can make timely and effective feedback to the changes in the external environment by forming a sensory system through a perception center, a sensor network, a signal processing unit and an execution unit, thereby greatly improving the ability to interact with the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a top view of the head knot of the present utility model.

[0018] Figure 2 Schematic diagram of the visual system of the present invention Figure 1 .

[0019] Figure 3 Schematic diagram of the visual system of the present invention Figure 2 .

[0020] Figure 4 Schematic diagram of the visual system of the present invention Figure 3 .

[0021] Figure 5 This is a schematic diagram of an eyeball of the present invention.

[0022] Figure 6 This is a schematic diagram of the eyeball decomposition of the present invention. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present invention through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the details in this specification may be modified or altered based on different perspectives and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features within these embodiments may be combined with one another, unless they conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. Therefore, the drawings only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0025] All directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, horizontal, vertical...) are only used to explain the relative position relationship, movement status, etc. between the various components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0026] Due to installation errors and other reasons, the parallel relationship referred to in the embodiments of the present invention may actually be an approximately parallel relationship, and the perpendicular relationship may actually be an approximately perpendicular relationship.

[0027] Example 1:

[0028] like Figures 1-6 As shown, the eye structure of the bionic service humanoid robot is installed on the head structure 1, including a sensory system and a visual system 3.

[0029] The sensory system includes a perception center, a sensor network, a signal processing unit and an execution unit. The sensor network signal is connected to the processing unit, the signal processing unit is connected to the perception center, and the perception center is connected to the execution unit.

[0030] The sensor network includes several sensors, including visual sensors, which capture visual information of the external environment, such as images. The sensor network transmits the captured information to the signal processing unit. The sensor network provides various information about the external environment and is the input source of the perception center.

[0031] The signal processing unit is used to receive the original signal from the sensor network, perform signal processing and preprocessing to enhance the accuracy and stability of the signal, and then transmit the processed signal to the perception center to provide the perception center with more reliable information.

[0032] The perception center, as the intelligent core of the humanoid robot, receives processed information, performs analysis and decision-making, and determines the state of the external environment and the required response to achieve perception of the surrounding environment and response to external stimuli. The perception center is connected to the visual system 3 to achieve overall perception and decision-making.

[0033] The Perception Center leverages high-performance processors from NVIDIA's Project GR00T and Isaac robotics platforms. Leveraging NVIDIA's unique SoC (system-on-chip) technology, it is optimized for the needs of complex robotic systems, enabling highly automated and intelligent operations. Specifically, it provides the following capabilities:

[0034] 1. High-performance computing support:

[0035] NVIDIA's SoC design is primarily designed to provide sufficient processing power to support the execution of AI algorithms and models in robots. This chip integrates a highly efficient GPU (graphics processing unit), enabling robots to quickly process visual and perceptual data and perform complex image recognition, object detection, and environmental understanding in real time. This is because GPUs are designed to process large amounts of data in parallel, making them ideal for deep learning tasks.

[0036] 2. Multimodal sensor fusion:

[0037] In humanoid robot applications, data from multiple sensors needs to be integrated and processed, including vision, hearing, touch, and position perception. NVIDIA's SoCs support this advanced sensor fusion, enabling robots to more accurately understand and adapt to their environment. For example, a robot can simultaneously process visual data from a camera and force feedback from sensors to achieve more precise object manipulation and navigation.

[0038] 3. Low latency and real-time response:

[0039] Humanoid robots require extremely low response times when performing tasks such as delivery, rescue, or collaborative work. NVIDIA SoCs ensure low latency and high-speed data processing by optimizing computational paths and improving data transmission efficiency. This allows robots to react quickly in dynamic and unpredictable environments.

[0040] 4. Energy efficiency:

[0041] Considering that humanoid robots need to operate for extended periods of time on battery power, energy efficiency becomes a key factor in SoC design. NVIDIA's chips maintain high performance while optimizing energy consumption through advanced manufacturing processes and power management technologies, extending the robot's operating time.

[0042] 5. Collaborative optimization of software and hardware:

[0043] The Isaac robotics platform includes not only the hardware SoC but also a complete software development kit (SDK), including a simulator, development tools, and pre-trained AI models. These software tools are tightly integrated with the SoC hardware, enabling developers to customize and optimize robot behavior for specific application scenarios, thereby improving development efficiency and robot performance.

[0044] The execution unit receives instructions from the perception center and performs corresponding actions or tasks, such as moving, operating external devices, etc.

[0045] The execution unit can adopt servo motor, electromagnetic brake, linear actuator, sensor and brake, etc.

[0046] The signal processing unit is the core processing unit for processing all sensory data, specifically including the central processing unit (CPU) and the graphics processing unit (GPU);

[0047] The central processing unit is responsible for processing program instructions, managing software operations and other computing tasks, while the graphics processing unit (GPU) is used for image analysis and machine vision, such as NVIDIA's Jetson series.

[0048] The central processing unit (CPU) can use conventional NVIDIA Jetson AGX Xavier, Intel Core i7-1185G7, AMD Ryzen 95900HX, Qualcomm Snapdragon 888, Apple M1, etc.

[0049] The graphics processing unit GPU can use conventional NVIDIA GeForce RTX 3080, AMD Radeon RX6800XT, Intel Iris Xe Graphics G7, Qualcomm Adreno660, Apple M1 GPU, etc.

[0050] The graphics processing unit and visual sensor constitute a visual video tracking system. The visual sensor captures visual information of the surrounding environment and transmits it to the graphics processing unit for analysis, recognition and tracking.

[0051] The graphics processing unit has a built-in image processing algorithm. The vision sensor captures continuous image frames and then uses the image processing algorithm to identify and extract the required image or object. The specific steps include:

[0052] Step S1: The visual sensor captures real-time images or videos of the environment and transmits them to the graphics processing unit;

[0053] Step S2: The image processing algorithm pre-processes the captured image;

[0054] Preprocessing includes image enhancement such as resizing, normalization, and denoising to improve the effect and efficiency of subsequent processing steps;

[0055] Step S3: Feature extraction: extracting specific information from the preprocessed image to help identify objects in the image;

[0056] Feature extraction methods include edge detection based on edge detection algorithms, corner detection based on corner detection algorithms, texture analysis based on texture analysis algorithms, and the like.

[0057] Step S4: object recognition;

[0058] Identify and classify objects in images using a specified machine learning model;

[0059] Machine learning models have been trained on large amounts of labeled data to recognize different types of objects;

[0060] Step S5: object positioning and tracking;

[0061] After object recognition, the specific location of the object in the image is determined based on the bounding boxes, which frame each recognized object in the image and locate it;

[0062] Step S6: output the result;

[0063] The graphics processing unit outputs the results of recognition and positioning to the perception center, which analyzes the results and determines whether to generate a corresponding decision. If so, it adjusts the parameters and executes step S2 to process the newly captured image. If not, it adjusts the model and calls a deep learning model (such as a convolutional neural network) and executes step S4.

[0064] Step S4 object recognition is achieved through machine learning and computer vision technology. Common methods include:

[0065] Object classification: Classify objects in an image into predefined categories.

[0066] Object Detection: Detecting object locations and bounding boxes in images.

[0067] Object tracking: Tracking the motion trajectory of a specific object in consecutive image frames.

[0068] The actions that can be triggered based on different images are determined by pre-defined programs or algorithms. For example, if the image recognition system detects a face, the robot's head may turn and face the face; if it detects a specific object, the robot may perform tasks or actions related to the object, such as grasping or moving it.

[0069] This embodiment proposes a mathematical model for an image recognition algorithm based on neural network multi-model feature fusion, integrating image features from multiple sensors to improve image recognition accuracy. Specifically:

[0070] Assume that the number of sensors is n, each sensor can capture images and extract image features. Let the image features captured by the i-th sensor be expressed as Where i∈{1, 2, ..., n}. These image features are integrated into a feature vector, represented as x∈R m ,in

[0071] A deep neural network is established for image recognition, and its structure is as follows:

[0072] h=σ(W1x+b1)

[0073] y=softmax(W2h+b2)

[0074] Among them, is the category probability distribution predicted by the model, h represents the hidden layer bias vector, W1∈R h×m and W2∈R k×h Represents the weight matrices from the input layer to the hidden layer and from the hidden layer to the output layer, R h×m Represents the weight matrix from the input layer to the hidden layer, R k×h Represents the weight matrix from the hidden layer to the output layer, b1∈R h and b2∈R k Represent the bias vectors of the hidden layer and the output layer respectively, k represents the bias vector of the output layer, R h Represents the weight matrix of the input layer, R k Represents the weight matrix of the hidden layer; σ() represents the activation function, and softmax() represents the softmax function.

[0075] To train the neural network, the cross entropy loss function is used:

[0076]

[0077] Among them, y is the category probability distribution predicted by the model, is the true category label, and j represents the category index.

[0078] Minimize the loss function by gradient descent, that is, update the weights and biases:

[0079]

[0080] Where α is the learning rate, are the gradients of the loss function with respect to the weight matrices W1, W2 and the bias vectors b1, b2 respectively.

[0081] Use a deep convolutional neural network (CNN) for image recognition. CNN contains multiple convolutional layers, pooling layers, and fully connected layers to extract image features and perform classification. Let the input of CNN be x and the output be y∈R k , represents the category prediction result of the image.

[0082] The output y of CNN can be expressed as:

[0083] h1=ReLU(W1x+b1)

[0084] h2=MaxPooling(h1)

[0085] h3=ReLU(W2h2+b2)

[0086] h4=MaxPooling(h3)

[0087] h5=ReLU(W3h4+b3)

[0088] h6=Flatten(h5)

[0089] y=Softmax(W4h6+b4)

[0090] Among them, b3 and b4 are bias vectors, W3 and W4 are weight matrices, ReLU() represents the rectified linear unit activation function, MaxPooling() represents the maximum pooling operation, Flatten() represents flattening multidimensional data into a one-dimensional vector, and Softmax() represents the softmax function.

[0091] The cross entropy loss function is used to measure the difference between the model prediction results and the true labels:

[0092]

[0093] Among them, y is the category probability distribution predicted by the model, is the true category label, k represents the number of categories, and j is the starting parameter value of the model.

[0094] Minimize the loss function by gradient descent, that is, update the weights and biases:

[0095]

[0096] Where α is the learning rate, and $ $represents the gradient of the loss function with respect to the weight matrix W1 and the bias vector b1 respectively.

[0097] Calculation example:

[0098] Assume that there are three sensors, which capture the color features, texture features and shape features of the image respectively. The output feature vector of each sensor is x1∈R 10 x1∈R10, x2∈R 15 x2∈R15 and x3∈R 20 x3∈R20.

[0099] The goal is to use a deep convolutional neural network to identify the category of the image, assuming there are 5 categories.

[0100] First, initialize the parameters of the neural network. Assume that the hidden layer contains 20 neurons and the learning rate is 0.01.

[0101] [W1∈R 20×(10+15+20) , W2∈R 5×20 ][b1∈R 20 , b2∈R 5 ]

[0102] Next, prepare the training data. Assume there are 1,000 examples, each with a one-hot encoding of the label. We will use stochastic gradient descent to update the parameters, using one example at a time.

[0103] Then, you can train and predict the model. Suppose you have completed 100 rounds of iterative training and want to make a classification prediction for a new image.

[0104] Finally, the prediction results will be output and the accuracy of the model will be calculated.

[0105] Calculation process:

[0106] Assume that the hidden layer contains 20 neurons and the learning rate is 0.01. The initialization parameters are as follows:

[0107] [W1∈R 20×45 , W2∈R 5×2 0\]\[b1∈R 20 , b2∈R 5 \]

[0108] Prepare training data: Assume there are 1000 samples, and the label of each sample is represented by one-hot encoding.

[0109] Use stochastic gradient descent to update parameters, training one sample at a time, and complete 100 rounds of training.

[0110] Based on the above premise, suppose there is a new image that needs classification prediction. The output result is category 3 with an accuracy of 80%.

[0111] The visual sensor may adopt conventional Sony IMX477 CMOS image sensor, OmniVision OV5670 CMOS image sensor, ON Semiconductor AR0144 CMOS image sensor, Samsung S5K4H7YX CMOS image sensor, Canon 5D Mark IV CMOS image sensor, etc.

[0112] The sensory system also includes a sensory central anti-electromagnetic interference system and a sensory central storage chip family. The sensory central anti-electromagnetic interference system implements electromagnetic interference (EMI) protection. The sensory central anti-electromagnetic interference system uses EMI shielding materials and covers sensitive components with conductive or magnetic materials to reduce the impact of external electromagnetic waves; the sensory central storage chip family is used to store memory and storage devices for data collected from various sensor networks, including random access memory (RAM): fast access memory for temporary storage of data in processing and solid-state drive (SSD): for long-term data storage, with fast read and write speeds and high durability.

[0113] The sensory central storage chip family can use conventional Samsung PM9A3 E1.S SSD, Western Digital WD Black SN850 NVMe SSD, SK Hynix Gold P31 NVMe SSD, Crucial P5 Plus NVMe SSD, Intel Optane SSD 905P U.2SSD, etc.

[0114] An inspection hatch 11 is provided on the rear side of the head structure 1, and the inspection hatch 11 is hinged to the head structure 1. This hinged structure allows the inspection hatch 11 to be opened and stay at a certain angle, which facilitates access to the internal structure of the head structure 1. The inspection hatch 11 facilitates the replacement and maintenance of the robot's sensory system. The design of the inspection hatch 11 allows technicians to easily replace or upgrade these components, as well as perform routine maintenance and fault diagnosis.

[0115] A memory chip insertion slot and a hatch 12 are also provided on the rear side of the skull structure 1. The hatch 12 can be detachably installed in the memory chip insertion slot, and the sensory central memory chip family is installed in the memory chip insertion slot.

[0116] The skull structure is provided with two groups of visual mounting holes that are symmetrical on the left and right. The visual system 3 is provided with two groups. The two groups of visual systems 3 are respectively arranged at the positions of the two groups of visual mounting holes. The two groups of visual systems 3 have the same structure. This embodiment describes the structure of one group of visual systems 3. The visual system 3 includes an eye shell 30 and an eyebrow structure 31, an eyelid structure 32 and an eyeball 33 installed in the eye shell 3. The eyebrow structure 31 is arranged above the visual mounting hole. The eyelid structure 32 and the eyeball 33 are installed in the visual mounting hole. The eyebrow structure 31 includes an eyebrow movable bracket 311 and two groups of eyebrow movable components 312. The two groups of eyebrows The movable components 312 are respectively connected to the eyebrow movable bracket 311, and the up and down movement of the eyebrow movable bracket 311 is realized through the eyebrow movable component 312. The eyelid structure 32 includes an upper eyelid bracket 321, an upper eyelid 322, a lower eyelid 323, a lower eyelid bracket 324 and an eyelid movable component. The eyelid movable component controls the opening and closing movement of the upper eyelid bracket 321 and the lower eyelid bracket 324. The eyeball 33 includes a main body assembly 331 and an eye movable component 332. The main body assembly 33 is movably installed in the middle of the eyelid structure 32. The eye movable component 332 controls the up and down movement and rotation of the main body assembly 331.

[0117] The eye housing 30 serves as an outer structure that provides protection and support for the internal components of the eye.

[0118] The eyebrow movable support 31, the upper eyelid support 321, and the lower eyelid support 324 are made of ABS material; the upper eyelid 322 and the lower eyelid 323 are made of high-temperature resistant silicone polymer, on which eyelashes can be installed.

[0119] The eyebrow movable assembly 312 includes a servo servo motor 3121 and an eyebrow connecting rod 3122. The upper end of the eyebrow connecting rod 3122 is connected to an upper movable joint 3123, and the lower end of the eyebrow connecting rod 3122 is connected to a lower movable joint 3124. The servo servo motor 3121 is installed on the head structure. The upper movable joint 3123 is connected to the servo servo motor 3121 through an eyebrow movable pin 3125, which can realize a rotation similar to a universal ball. The eyebrow movable bracket 311 is provided with an eyebrow fixing pin, and the lower movable joint 3124 is connected to the eyebrow fixing pin.

[0120] The servo motor 3121 starts to control the rotation of the eyebrow movable pin 3125. When the eyebrow movable pin 3125 rotates from the bottom to the top, it drives the eyebrow connecting rod 3122 to move upward, and then drives the eyebrow movable bracket 311 to move upward. When the eyebrow movable pin 3125 rotates from the top to the bottom, it drives the eyebrow connecting rod 3122 to move downward, and then drives the eyebrow movable bracket 311 to move downward, thereby controlling the up and down movement of the eyebrow movable bracket 311.

[0121] The two sets of eyebrow movable supports 311 of the two sets of visual systems 3 can perform synchronous or asynchronous movement through the drive of the eyebrow movable component 312, thereby realizing the expression of various emotions by the robot.

[0122] In this embodiment, the eyebrow movable bracket 311 is controlled by two groups of eyebrow movable components 312 to perform corresponding movements. The eyebrow movable component 312 is connected to the execution unit. After receiving the eyebrow movable bracket activity trigger instruction and processing it, the execution unit sends an eyebrow movable bracket control signal to the eyebrow movable component 312, thereby controlling the movement of the eyebrow movable bracket 311.

[0123] The eyebrow movable bracket activity triggering instruction is the eyebrow movable bracket up and down movement triggering instruction and the asynchronous movement instruction of the two groups of eyebrow movable bracket activities at the left and right positions, including:

[0124] Visual recognition commands: When a camera or other visual sensor detects a user's eye expression or gesture, it triggers the up and down movement of the right eyebrow based on the user's eye gaze or gesture. For example, when the user raises their eyebrows upward or frowns downward, the corresponding movement of the right eyebrow is triggered.

[0125] Voice interaction commands: When the humanoid robot has a voice conversation with the user, the up and down movement of the right eyebrow is triggered according to the content and direction of the voice command to express the robot's attention or response action.

[0126] The asynchronous motion triggering instructions for the two sets of eyebrow movable bracket activities include

[0127] Visual tracking commands: Track the user's head movements or facial expressions through a camera or other visual sensor. When the user's head or eyes turn left or right, the right eyebrow is triggered to move slightly left or right to simulate a person's eye gaze or facial expression changes.

[0128] Voice interaction commands: When the humanoid robot has a voice conversation with the user, it triggers left and right micro-movements of the right eyebrow based on the content and directionality of the voice command to express the robot's attention or response.

[0129] The upper eyelid 322 is fixed on the upper eyelid bracket 321, and the lower eyelid 323 is fixed on the lower eyelid bracket 324. The upper eyelid bracket 321 and the lower eyelid bracket 324 are respectively provided with tooth ends at the left and right ends. The two tooth ends are coaxial and meshingly arranged inside and outside. When one of the tooth ends is driven to rotate, the upper eyelid bracket 321 and the lower eyelid bracket 324 realize opposite rotation due to their meshing movement, which can make the upper eyelid bracket 321 and the lower eyelid bracket 324 open and close to realize blinking operation, and the opening and closing frequency can be set according to needs.

[0130] The eyelid movable component provides power for the eyelid movable component. Specifically, the eyelid movable component includes an eyelid opening and closing rotating shaft 325 and a servo servo. The servo servo is installed on the head structure. The output shaft of the servo servo is fixedly connected to the eyelid opening and closing rotating shaft 325. The eyelid opening and closing rotating shaft 325 is fixedly connected to the tooth end located on the inner side. When the servo servo is started, power is transmitted to the inner tooth end through the eyelid opening and closing rotating shaft 325, thereby realizing the opening and closing movement of the upper eyelid bracket 321 and the lower eyelid bracket 324.

[0131] The two sets of eyelid structures 32 of the two sets of visual systems 3 can perform synchronous opening and closing movements or asynchronous opening and closing movements through the driving of the eyelid movable components.

[0132] In this embodiment, the upper eyelid support 321 and the lower eyelid support 324 are controlled by the eyelid movable component to perform corresponding movements. The servo servo is connected to the execution unit. After receiving the eyelid support trigger instruction and processing it, the execution unit sends the eyelid support control signal to the servo servo, thereby controlling the movement of the eyelid support.

[0133] The eyelid support triggering instructions include instructions for triggering the opening and closing movements of the upper eyelid support 321 and the lower eyelid support 324;

[0134] The upper eyelid support 321 and the lower eyelid support 324 opening and closing movement triggering instructions include

[0135] Visual recognition commands: When a camera or other visual sensor detects a user's eye expression or gesture, the right upper eyelid bracket opens and closes according to the user's eye state. For example, when the user closes or opens their eyes, the corresponding action of the right upper eyelid bracket is triggered;

[0136] Emotion simulation instructions: Analyze emotional signals in the environment based on the emotion recognition algorithm. For example, when the user's tiredness, surprise, or joy is detected, the opening and closing movement of the right upper eyelid bracket is triggered to simulate the corresponding changes in eye expression.

[0137] The main body assembly 331 includes an eyeball-shaped shell 3315, an eyeball-shaped shell back cover 3316 and a back cover wire collection slot locking nut 3317. The visual sensor 3311, a variable focus movable motor seat 3312, an integrated circuit board 3313 and a rubber ring for fixing the connecting rod seat 3314 are fixed in sequence inside the eyeball-shaped shell 3315. The eyeball-shaped shell back cover 3316 is fixed to the end of the eyeball-shaped shell 3315 and is locked by the back cover wire collection slot locking nut 3317.

[0138] The visual sensor 3311 can use a micro high-definition camera with functions such as automatic exposure / auto focus / auto white balance / auto fill light to capture visual information of the surrounding environment.

[0139] The visual sensor 3311 is part of the visual video tracking system and transmits the captured high-definition video and images to the graphics processing unit for processing.

[0140] The eye movement component 332 is used to control the movement of the eyeball so that it can rotate in the horizontal and vertical directions, thereby changing the direction of sight. The eye movement component 332 includes an eyeball support rod 3320, an eyeball connecting rod 3322 and a servo servo 3325. The eyeball support rod 3320 is located in the middle and connects the eyeball 33 and the eyeball connecting block 3324. The eyeball connecting block 3324 is installed on the skull structure. The servo servo 3325 is installed on the eyeball connecting block 3324. Active joints are provided at both ends of the eyeball connecting rod 3322. The active joint at one end is connected to the eyeball contour shell 3315 through a fixed pin, and the other end is connected to the servo servo 3325 through a active pin. A group of servo servos 3325 controls the movement of a group of eyeball connecting rods 3322.

[0141] When several servo servos 3325 control the main body assembly 331 to move forward, and the remaining servo servos 3325 control the main body assembly 331 to move backward, the main body assembly 331 can be rotated up or down. This embodiment controls different servo servos 3325 to drive different eyeball connecting rods 3322 to achieve horizontal and vertical rotation of the eyeball, thereby changing the direction of the line of sight.

[0142] In this embodiment, when the robot detects a person or object entering its surroundings, the visual tracking system begins capturing images and transmits them to the perception center for analysis. If the perception center determines that the line of sight needs to be adjusted to track the target, it sends corresponding control instructions to the eye movement component 332 to trigger eye movement.

[0143] Eye movement triggering mainly includes target detection, target importance assessment, target movement analysis, and the priority and requirements of the current task. Specifically, it includes the following steps:

[0144] Step A1. Target detection and confirmation:

[0145] 1.1. Detection: The visual video tracking system detects people or objects in the image;

[0146] 1.2. Confirmation: Confirm whether the detected object meets the tracking conditions and characteristics, such as a specific shape, color, or known markers.

[0147] Step A2. Goal Importance and Priority Assessment:

[0148] 2.1. Importance: Assessing the importance of a target depends on the specific requirements of the task (e.g., security monitoring, interactive tasks, or important objects in a specific scenario).

[0149] 2.2 Priority: In a multi-target environment, determine which targets have higher tracking priority based on their dynamic behavior, relevance to the task, or preset rules.

[0150] Step A3. Target dynamic analysis:

[0151] 3.1 Motion tracking: Analyze the target's trajectory to determine whether it is moving, its speed and direction.

[0152] 3.2 Predicting future position: Use a motion estimation model to predict the target’s future position to determine how to adjust the line of sight most effectively.

[0153] Step A4. Task and environment requirements:

[0154] 4.1 Task Requirements: The robot decides whether to adjust its line of sight based on the current task requirements (e.g., whether it needs to continuously monitor an area or object) or environmental changes (e.g., other moving objects, light changes, etc.).

[0155] Step A5. Eyesight adjustment strategy:

[0156] 5.1 Instant Adjustment: If the target's position or speed does not match the predetermined tracking strategy, immediately adjust the sight direction to keep the target in the center of the field of view or at an appropriate position.

[0157] 5.2 Predictive Adjustment: Based on the prediction model of target motion, adjust the gaze direction in advance to reduce reaction delay and improve tracking continuity and accuracy.

[0158] The trigger instructions may include

[0159] Visual tracking commands: A camera or other visual sensor captures the user's head movements and facial expressions in real time, triggering the synchronized movement of the left and right eyeballs based on the user's head movements. When the user raises their head, lowers it, or turns left or right, the left and right eyeballs move accordingly, simulating changes in a person's gaze.

[0160] Gesture recognition commands: The humanoid robot is equipped with gesture recognition technology that triggers eye movements based on the user's hand movements. For example, when the user points a finger in a specific direction, the eye moves in that direction.

[0161] Context-aware instructions: Using environmental perception technologies, such as sound sensors or depth cameras, the robot detects sounds or the location of objects in the surrounding environment, triggering asynchronous movements of the left and right eyeballs. For example, when a humanoid robot detects that a sound is coming from the left, the left eyeball may turn left while the right eyeball remains stationary, simulating the direction of human attention.

[0162] Emotional simulation commands: Analyze the user's emotional changes based on an emotion recognition algorithm and trigger asynchronous eye movements based on these changes. For example, if the user is detected to be anxious or surprised, the left and right eyeballs may move asynchronously up and down and left and right to express the robot's response to the user's emotions.

[0163] When the robot needs to change the direction of its sight, the perception center will analyze the current environment and determine the target position that needs to be adjusted. Specifically, based on the target position and the current position of the eyeball, the execution unit will send a corresponding control signal to the eye movement component 332 to adjust the direction and position of the eyeball to align with the target.

[0164] The robot needs to change the direction of its sight line, which is related to target tracking, task requirements and environmental changes.

[0165] Target tracking: Deviation of target position detected by vision sensors from the expected or planned path;

[0166] Task requirements: For example, the user needs to change their line of sight to adapt to changes in the environment or focus when navigating, avoiding obstacles, or interacting with people.

[0167] Environmental changes: Changes in the environment may require the robot to adjust its line of sight to gain more information or better understand its surroundings.

[0168] The specific steps include:

[0169] Step B1: Determination of target location;

[0170] The specific location or coordinates of the target are usually determined by the following steps:

[0171] 1.1. Image Capture: First, the vision sensor captures an image of the current field of view;

[0172] 1.2. Image processing: Identify objects in the image through the signal processing unit;

[0173] 1.3. Object localization: Locating the position of an object in three-dimensional space based on a deep learning model;

[0174] 1.4. Coordinate transformation: Based on the robot's own position and orientation coordinates, the coordinates of the target object are transformed into a position in the global coordinate system;

[0175] Step B2: Adjust eye position and sight direction;

[0176] The adjustment process includes:

[0177] 2.1. Error calculation: Calculate the deviation between the target position and the current line of sight center;

[0178] 2.2. Motion planning: Calculate the angle and direction of eye movement based on the deviation;

[0179] 2.3. Perform adjustment: By controlling the eye movement component 332, adjust the sight direction so that the target is located in the center of the visual field or other predetermined position.

[0180] In this embodiment, the voice instructions and voice interaction instructions are captured by the auditory sensor, processed by the signal processing unit, and sent to the execution unit by the perception center; the visual tracking / recognition instructions and emotion simulation instructions are captured by the visual sensor, processed by the signal processing unit, and sent to the execution unit by the perception center; and the emotion expression instructions are sent from the perception center to the execution unit.

[0181] Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

Claims

1. The eye structure of a bionic service-type humanoid robot is installed on the head structure and is characterized by: It includes a sensory system and a visual system connected to the sensory system. The visual system includes an eyebrow structure, an eyelid structure and an eyeball. The eyebrow structure, the eyelid structure and the eyeball receive control signals from the sensory system to perform corresponding movements. The eyebrow structure includes two groups of eyebrow movable components. The eyebrow movable components include a servo servo and an eyebrow connecting rod. The upper end of the eyebrow connecting rod is connected to an upper movable section, and the lower end of the eyebrow connecting rod is connected to a lower movable section. The servo servo is installed on the skull structure. The upper movable section is connected to the servo servo through an eyebrow movable pin. The eyebrow movable bracket is provided with an eyebrow fixing pin, and the lower movable section is connected to the eyebrow fixing pin.

2. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The eyebrow structure comprises an eyebrow movable support, and two groups of eyebrow movable components are respectively connected to the eyebrow movable support.

3. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The eyelid structure includes an upper eyelid support, an upper eyelid, a lower eyelid, a lower eyelid support and an eyelid movable component, and the eyelid movable component is used to control the opening and closing movements of the upper eyelid support and the lower eyelid support.

4. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The eyeball includes a body assembly and an eye movable component. The body assembly is movably installed in the middle of the eyelid structure, and the up and down movement and rotation of the body assembly are controlled by the eye movable component.

5. The eye structure of the bionic service humanoid robot according to claim 3, characterized in that: The upper eyelid is fixed on the upper eyelid bracket, and the lower eyelid is fixed on the lower eyelid bracket. The upper eyelid bracket and the lower eyelid bracket are respectively provided with tooth ends at the left and right ends, and the two tooth ends are coaxial and meshingly arranged inside and outside.

6. The eye structure of the bionic service humanoid robot according to claim 5, characterized in that: The eyelid movable component includes an eyelid opening and closing rotating shaft, the output shaft of the servo steering engine is fixedly connected to the eyelid opening and closing rotating shaft, and the eyelid opening and closing rotating shaft is fixedly connected to the tooth end located on the inner side.

7. The eye structure of the bionic service humanoid robot according to claim 4, characterized in that: The main body assembly includes an eyeball-shaped shell, an eyeball-shaped shell back cover and a back cover wire collection slot locking nut. A visual sensor, a variable focus movable motor seat, an integrated circuit board and a rubber ring for fixing a connecting rod seat are fixed in sequence inside the eyeball-shaped shell. The eyeball-shaped shell back cover is fixed to the end of the eyeball-shaped shell and is locked by the back cover wire collection slot locking nut.

8. The eye structure of the bionic service humanoid robot according to claim 7, characterized in that: The eye movement component includes an eyeball support rod, an eyeball connecting rod and a servo servo. The eyeball support rod is located in the middle and connects the eyeball and the eyeball connecting block. The eyeball connecting block is installed on the skull structure. The servo servo is installed on the eyeball connecting block. Active joints are provided at both ends of the eyeball connecting rod. The active joint at one end is connected to the eyeball contour shell through a fixed pin, and the other end is connected to the servo servo through a active pin. A group of servo servos controls the movement of a group of eyeball connecting rods.

9. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The visual system also includes an eye shell, and the eyebrow structure, eyelid structure and eyeball are installed inside the eye shell.

Images