Eye structure of bionic service type humanoid robot

Abstract

The invention discloses a bionic service type humanoid robot eye structure which is installed on a head structure and comprises a sensory system and a visual system, the sensory system comprises a sensing center, a sensor network, a signal processing unit and an execution unit, the sensor network is connected with the signal processing unit, the signal processing unit is connected with the sensing center, and the execution unit is connected with the sensing center. The sensing center is connected with the execution unit, the execution unit is connected with the visual system, the sensing center, the sensor network, the signal processing unit and the execution unit form a sensory system, timely and effective feedback can be conducted on external environment changes, and the interaction capacity with the outside is greatly improved.

Description

Technical Field

[0001] The invention belongs to the field of robot bionics and relates to the eye structure of a bionic service-type humanoid robot. Background Art

[0002] A "humanoid robot" can be defined as a robot that has certain attributes of human appearance and functionality (e.g., torso, head, arms, legs), the ability to communicate verbally with humans using speech recognition and sound synthesis, etc. This type of robot aims to reduce the cognitive distance between humans and machines.

[0003] Existing "humanoid robots" have limited ability to interact with the outside world and are unable to provide timely feedback based on changes in the external environment. Summary of the invention

[0004] In order to overcome at least one deficiency of the prior art, the present invention provides an eye structure of a bionic service-type humanoid robot.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: the eye structure of the bionic service humanoid robot is installed in the head structure, including a sensory system and a visual system, the sensory system includes a perception center, a sensor network, a signal processing unit and an execution unit, the sensor network is connected to the signal processing unit, the signal processing unit is connected to the perception center, the perception center is connected to the execution unit, and the execution unit is connected to the visual system;

[0006] The sensor network includes several visual sensors that capture visual information of the external environment;

[0007] The signal processing unit is used to receive the original signal from the sensor network, perform signal processing and preprocessing, and transmit the processed signal to the perception center;

[0008] The perception center receives the processed signals, performs analysis and decision-making, and obtains the status of the external environment and the required response;

[0009] The execution unit receives the triggering instruction from the perception center and executes the corresponding action or task;

[0010] The visual system receives the control signal sent by the execution unit and performs corresponding movements.

[0011] Furthermore, the signal processing unit includes a graphics processing unit, and the graphics processing unit and the visual sensor constitute a visual video tracking system. The visual sensor captures visual information of the surrounding environment and transmits it to the graphics processing unit for analysis, recognition and tracking. The graphics processing unit has a built-in image processing algorithm, and the visual sensor captures continuous image frames. The required image or object is recognized and extracted through the image processing algorithm, which specifically includes the following steps:

[0012] Step S1: The visual sensor captures real-time images or videos in the environment and transmits them to the graphics processing unit;

[0013] Step S2: The image processing algorithm pre-processes the captured image;

[0014] Step S3: Feature extraction: extracting specific information from the preprocessed image;

[0015] The feature extraction methods include edge detection based on edge detection algorithm, corner detection based on corner detection algorithm, and texture analysis based on texture analysis algorithm;

[0016] Step S4: object recognition;

[0017] Identify and classify objects in images using a set machine learning model;

[0018] Step S5: object positioning and tracking;

[0019] After object recognition, the specific location of the object in the image is determined based on the bounding box, which frames each recognized object in the image to locate it;

[0020] Step S6: output the result;

[0021] The graphics processing unit outputs the results of recognition and positioning to the perception center, which analyzes the results to determine whether to generate a corresponding decision. If so, it adjusts the parameters and executes step S2 to process the newly captured image. If not, it adjusts the model, calls the deep learning model, and executes step S4.

[0022] Furthermore, the sensory system also includes a sensory central anti-electromagnetic interference system and a sensory central storage chip family. The sensory central anti-electromagnetic interference system realizes electromagnetic interference protection. The sensory central anti-electromagnetic interference system uses conductive or magnetic materials to cover sensitive components. The sensory central storage chip family is used to store memory and storage devices for storing data collected from the sensor network, including random access memory, fast access memory and solid-state hard disk. The fast access memory is used to temporarily store data in processing, and the solid-state hard disk is used for long-term data storage.

[0023] Furthermore, the visual system includes an eye shell and an eyebrow structure, an eyelid structure and an eyeball installed in the eye shell, the eyebrow structure includes an eyebrow movable bracket and two groups of eyebrow movable components, the two groups of eyebrow movable components are respectively connected to the eyebrow movable bracket, the eyelid structure includes an upper eyelid bracket, an upper eyelid, a lower eyelid, a lower eyelid bracket and an eyelid movable component, the eyelid movable component controls the opening and closing movements of the upper eyelid bracket and the lower eyelid bracket, the eyeball includes a body assembly and an eye movable component, the body assembly is movably installed in the middle of the eyelid structure, and the eye movable component controls the up and down movement and rotation of the body assembly.

[0024] Furthermore, the eyebrow movable component includes a servo servo motor and an eyebrow connecting rod, the upper end of the eyebrow connecting rod is connected to an upper movable joint, the lower end of the eyebrow connecting rod is connected to a lower movable joint, the servo servo motor is installed on the skull structure, the upper movable joint is connected to the servo servo motor through an eyebrow movable pin, the eyebrow movable bracket is provided with an eyebrow fixing pin, and the lower movable joint is connected to the eyebrow fixing pin.

[0025] Furthermore, the eyebrow movable support is controlled by two groups of eyebrow movable components to perform corresponding movements. The eyebrow movable components are connected to the execution unit. After receiving the eyebrow movable support activity trigger instruction and processing it, the execution unit sends an eyebrow movable support control signal to the eyebrow movable component to control the eyebrow movable support structure to move synchronously or asynchronously.

[0026] Furthermore, the main body assembly includes an eyeball-shaped shell, an eyeball-shaped shell back cover and a back cover wire collection slot locking nut. A visual sensor, a variable-focus movable motor seat, an integrated circuit board and a rubber ring for fixing a connecting rod seat are fixed in sequence inside the eyeball-shaped shell. The eyeball-shaped shell back cover is fixed to the end of the eyeball-shaped shell and is locked by the back cover wire collection slot locking nut.

[0027] Furthermore, when the robot detects that a person or object has entered its surroundings, the visual video tracking system captures the image and transmits it to the perception center for analysis. If the perception center determines that the line of sight needs to be adjusted to track the target, the corresponding control instruction is sent to the eye movement component 332 to trigger the eye movement:

[0028] The triggering of eye movements includes target detection, target importance assessment, analysis of target movement, and the priority and requirements of the current task, which specifically includes the following steps:

[0029] Step A1. Target detection and confirmation:

[0030] 1.1. Detection: The visual video tracking system detects people or objects in the image;

[0031] 1.2. Confirmation: Confirm whether the detected object meets the tracking conditions and characteristics;

[0032] Step A2. Goal importance and priority assessment:

[0033] 2.1. Importance: Evaluate the importance of the goal, which depends on the specific requirements of the task;

[0034] 2.2 Priority: In a multi-target environment, determine which targets have a higher tracking priority based on the target's dynamic behavior, task relevance, or preset rules;

[0035] Step A3. Target dynamic analysis:

[0036] 3.1 Motion tracking: Analyze the target's motion trajectory to determine whether it is moving, its speed and direction;

[0037] 3.2 Predicting future position: Use the motion estimation model to predict the target’s future position;

[0038] Step A4. Task and environment requirements:

[0039] 4.1 Task requirements: Decide whether to adjust the line of sight based on the robot’s current task requirements or environmental changes;

[0040] Step A5. Sight adjustment strategy:

[0041] 5.1 Instant adjustment: If the position or speed of the target does not match the predetermined tracking strategy, immediately adjust the sight direction to keep the target in the center of the field of view or in an appropriate position;

[0042] 5.2 Predictive adjustment: Adjust the sight direction in advance based on the prediction model of target motion.

[0043] Further, when the robot needs to change the direction of sight, the perception center analyzes the current environment, determines the target position that needs to be adjusted according to the target position and the current position of the eyeball, and the execution unit sends a corresponding control signal to the eye movement component 332 to adjust the direction and position of the eyeball, which specifically includes the following steps:

[0044] Step B1: Determination of target location;

[0045] The specific location or coordinates of the target are usually determined by the following steps:

[0046] 1.1. Image Capture: First, the visual sensor captures the image of the current field of view;

[0047] 1.2. Image processing: Identify objects in the image through the signal processing unit;

[0048] 1.3. Object positioning: Locate the position of an object in three-dimensional space based on a deep learning model;

[0049] 1.4. Coordinate transformation: According to the coordinates of the robot's own position and orientation, the coordinates of the target object are transformed into the position in the global coordinate system;

[0050] Step B2: Adjust the eye position and sight direction;

[0051] The adjustment process includes:

[0052] 2.1. Error calculation: Calculate the deviation between the target position and the current line of sight center;

[0053] 2.2. Motion planning: Calculate the angle and direction of eye movement based on the deviation;

[0054] 2.3. Perform adjustment: By controlling the eye movement component 332, adjust the line of sight so that the target is located in the center of the visual field or other predetermined position.

[0055] In summary, the present invention is beneficial in that:

[0056] The present invention forms a sensory system through a perception center, a sensor network, a signal processing unit and an execution unit, which can give timely and effective feedback to changes in the external environment, greatly improving the ability to interact with the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a top view of the head knot of the present invention.

[0058] Figure 2 Schematic diagram of the visual system of the present invention Figure 1 .

[0059] Figure 3 Schematic diagram of the visual system of the present invention Figure 2 .

[0060] Figure 4 Schematic diagram of the visual system of the present invention Figure 3 .

[0061] Figure 5 It is a schematic diagram of an eyeball of the present invention.

[0062] Figure 6 It is a schematic diagram of the decomposition of the eyeball of the present invention. DETAILED DESCRIPTION

[0063] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0064] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0065] All directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, horizontal, vertical...) are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0066] Due to installation errors and other reasons, the parallel relationship referred to in the embodiments of the present invention may actually be an approximately parallel relationship, and the vertical relationship may actually be an approximately vertical relationship.

[0067] Embodiment 1:

[0068] like Figure 1-Figure 6 As shown, the eye structure of the bionic service humanoid robot is installed on the skull structure 1, including a sensory system and a visual system 3.

[0069] The sensory system includes a perception center, a sensor network, a signal processing unit and an execution unit. The sensor network signal is connected to the processing unit, the signal processing unit is connected to the perception center, and the perception center is connected to the execution unit.

[0070] The sensor network includes several sensors, including visual sensors, which capture visual information of the external environment, such as images. The sensor network transmits the captured information to the signal processing unit. The sensor network provides various information about the external environment and is the input source of the perception center.

[0071] The signal processing unit is used to receive the original signal from the sensor network, perform signal processing and preprocessing, enhance the accuracy and stability of the signal, and then transmit the processed signal to the perception center to provide more reliable information for the perception center.

[0072] As the intelligent core of the humanoid robot, the perception center receives processed information, performs analysis and decision-making, and determines the state of the external environment and the required response to achieve perception of the surrounding environment and response to external stimuli. The perception center is connected to the visual system 3 to achieve overall perception and decision-making.

[0073] The Perception Center uses high-performance processors from NVIDIA's Project GR00T and Isaac robotics platforms, and uses NVIDIA's unique SoC (system-on-chip) technology to optimize the design for complex robotic systems to achieve highly automated and intelligent operations. Specifically, it has the following functions:

[0074] 1. High performance computing support:

[0075] NVIDIA's SoC design is mainly designed to provide enough processing power to support the operation of AI algorithms and models in robots. This chip integrates an efficient GPU (graphics processing unit), which enables robots to quickly process visual and perceptual data and perform complex image recognition, object detection and environmental understanding in real time. This is because GPUs are designed for parallel processing of large amounts of data and are very suitable for deep learning tasks.

[0076] 2. Multimodal sensor fusion:

[0077] In humanoid robot applications, data from multiple sensors needs to be integrated and processed, including vision, hearing, touch, and position perception. NVIDIA's SoC supports this advanced sensor fusion, enabling robots to more accurately understand and adapt to their environment. For example, a robot can simultaneously process visual data from a camera and force feedback from a sensor to achieve more precise object manipulation and navigation.

[0078] 3. Low latency and real-time response:

[0079] Humanoid robots require extremely low response times when performing tasks such as delivery, rescue, or collaborative work. NVIDIA SoC ensures low latency and high-speed data processing capabilities by optimizing computing paths and improving data transmission efficiency. This allows robots to respond quickly in dynamic and unpredictable environments.

[0080] 4. Energy efficiency:

[0081] Considering that humanoid robots need to operate for long periods of time on battery power, energy efficiency becomes a key factor when designing SoCs. While maintaining high performance, NVIDIA's chips also optimize energy consumption through advanced manufacturing processes and power management technologies, extending the robot's operating time.

[0082] 5. Collaborative optimization of software and hardware:

[0083] The Isaac robot platform includes not only hardware SoC, but also a complete software development kit (SDK), including simulators, development tools, and pre-trained AI models. These software tools are tightly integrated with the SoC hardware, enabling developers to customize and optimize robot behavior for specific application scenarios, thereby improving development efficiency and robot performance.

[0084] The execution unit receives instructions from the perception center and performs corresponding actions or tasks, such as moving, operating external devices, etc.

[0085] The execution unit can adopt servo motor, electromagnetic brake, linear actuator, sensor and brake, etc.

[0086] The signal processing unit is the core processing unit that processes all sensory data, and specifically includes the central processing unit (CPU) and the graphics processing unit (GPU);

[0087] The central processing unit is responsible for processing program instructions, managing software operations and other computing tasks, and the graphics processing unit (GPU) is used for image analysis and machine vision, such as NVIDIA's Jetson series.

[0088] The central processing unit (CPU) can use conventional NVIDIA Jetson AGX Xavier, Intel Core i7-1185G7, AMD Ryzen 9 5900HX, Qualcomm Snapdragon888, Apple M1, etc.

[0089] The graphics processing unit GPU can use conventional NVIDIA GeForce RTX 3080, AMD Radeon RX6800XT, Intel Iris Xe Graphics G7, Qualcomm Adreno660, Apple M1 GPU, etc.

[0090] The graphics processing unit and the visual sensor constitute a visual video tracking system. The visual sensor captures the visual information of the surrounding environment and transmits it to the graphics processing unit for analysis, recognition and tracking.

[0091] The graphics processing unit has a built-in image processing algorithm. The visual sensor captures continuous image frames and then uses the image processing algorithm to identify and extract the required image or object. The specific steps include:

[0092] Step S1: The visual sensor captures real-time images or videos in the environment and transmits them to the graphics processing unit;

[0093] Step S2: The image processing algorithm pre-processes the captured image;

[0094] Preprocessing includes resizing, normalization, denoising, etc. to enhance the image to improve the effect and efficiency of subsequent processing steps;

[0095] Step S3: Feature extraction: extracting specific information from the preprocessed image can help identify the objects in the image;

[0096] The feature extraction methods include edge detection based on edge detection algorithm, corner detection based on corner detection algorithm, texture analysis based on texture analysis algorithm, etc.

[0097] Step S4: object recognition;

[0098] Identify and classify objects in images using a given machine learning model;

[0099] Machine learning models have been trained on large amounts of labeled data to recognize different types of objects;

[0100] Step S5: object positioning and tracking;

[0101] After object recognition, the specific location of the object in the image is determined based on the bounding boxes, which frame each recognized object in the image to locate it;

[0102] Step S6: output the result;

[0103] The graphics processing unit outputs the results of recognition and positioning to the perception center, which analyzes the results to determine whether to generate a corresponding decision. If so, it adjusts the parameters and executes step S2 to process the newly captured image. If not, it adjusts the model and calls a deep learning model (such as a convolutional neural network) to execute step S4.

[0104] Step S4 object recognition is achieved through machine learning and computer vision technology. Common methods include:

[0105] Object classification: Classify objects in an image into predefined categories.

[0106] Object Detection: Detecting object locations and bounding boxes in an image.

[0107] Object tracking: Tracking the motion trajectory of a specific object in consecutive image frames.

[0108] The actions that can be triggered based on different images depend on pre-defined programs or algorithms. For example, if the image recognition system detects a face, the robot head may turn and face the face; if a specific object is detected, the robot may perform tasks or actions related to the object, such as grasping, moving, etc.

[0109] This embodiment proposes a mathematical model of an image recognition algorithm based on neural network multi-model feature fusion, integrating image features from multiple sensors to improve the accuracy of image recognition. Specifically:

[0110] Assuming the number of sensors is n, each sensor can capture images and extract image features. Let the image features captured by the i-th sensor be expressed as Where i∈{1, 2, …, n}. These image features are integrated into a feature vector, represented as x∈R m ,in

[0111] A deep neural network is established for image recognition, and its structure is as follows:

[0112] h=σ(W1x+b1)

[0113] y=softmax(W2h+b2)

[0114] Among them, is the category probability distribution predicted by the model, h represents the hidden layer bias vector, W1∈R h×m and W2∈R k×h Represents the weight matrices from the input layer to the hidden layer and from the hidden layer to the output layer, R h×m Represents the weight matrix from the input layer to the hidden layer, R k×h Represents the weight matrix from the hidden layer to the output layer, b1∈R h and b2∈R k Represent the bias vectors of the hidden layer and the output layer respectively, k represents the bias vector of the output layer, R h represents the weight matrix of the input layer, R k represents the weight matrix of the hidden layer; σ() represents the activation function, and softmax() represents the softmax function.

[0115] To train the neural network, the cross entropy loss function is used:

[0116]

[0117] Among them, y is the category probability distribution predicted by the model, is the true category label, and j represents the category index.

[0118] Minimize the loss function by gradient descent, that is, update the weights and biases:

[0119]

[0120] Among them, α is the learning rate, They are the gradients of the loss function with respect to the weight matrices W1, W2 and the bias vectors b1, b2 respectively.

[0121] Use a deep convolutional neural network (CNN) for image recognition. CNN contains multiple convolutional layers, pooling layers, and fully connected layers to extract image features and perform classification. Let the input of CNN be x and the output be y∈R k , represents the category prediction result of the image.

[0122] The output y of CNN can be expressed as:

[0123] h1=ReLU(W1x+b1)

[0124] h2=MaxPooling(h1)

[0125] h3=ReLU(W2h2+b2)

[0126] h4=MaxPooling(h3)

[0127] h5=ReLU(W3h4+b3)

[0128] h6=Flatten(h5)

[0129] y=Softmax(W4h6+b4)

[0130] Among them, b3 and b4 are bias vectors, W3 and W4 are weight matrices, ReLU() represents the rectified linear unit activation function, MaxPooling() represents the maximum pooling operation, Flatten() represents flattening multidimensional data into a one-dimensional vector, and Softmax() represents the softmax function.

[0131] The cross entropy loss function is used to measure the difference between the model prediction and the true label:

[0132]

[0133] Among them, y is the category probability distribution predicted by the model, is the true category label, k represents the number of categories, and j is the starting parameter value of the model.

[0134] Minimize the loss function by gradient descent, that is, update the weights and biases:

[0135]

[0136] Among them, α is the learning rate, and They represent the gradients of the loss function with respect to the weight matrix W1 and the bias vector b1 respectively.

[0137] Calculation example:

[0138] Assume that there are three sensors, capturing the color features, texture features, and shape features of the image respectively. The output feature vector of each sensor is x1∈R 10 x1∈R10、x2∈R 15 x2∈R15 and x3∈R 20 x3∈R20.

[0139] The goal is to use a deep convolutional neural network to identify the category of the image, assuming there are 5 categories.

[0140] First, initialize the parameters of the neural network. Assume that the hidden layer contains 20 neurons and the learning rate is 0.01.

[0141] [W1∈R 20×(10+15+20) , W2∈R 5×20 ][b1∈R 20 , b2∈R 5 ]

[0142] Next, prepare the training data. Assume there are 1000 samples, and the label of each sample is represented by one-hot encoding. Stochastic gradient descent will be used to update the parameters, using one sample at a time.

[0143] Then, you can train and predict the model. Suppose you have completed 100 rounds of iterative training and want to make a classification prediction for a new image.

[0144] Finally, the prediction results will be output and the accuracy of the model will be calculated.

[0145] Operation process:

[0146] Assume that the hidden layer contains 20 neurons and the learning rate is 0.01. The initialization parameters are as follows:

[0147] [W1∈R 20×45 ,W2∈R 5×20 ], [b1∈R 20 ,b2∈R 5 ]

[0148] Prepare training data: Assume there are 1000 samples and the label of each sample is represented by one-hot encoding.

[0149] Use stochastic gradient descent to update parameters, using one sample at a time for training. Complete 100 rounds of iterative training.

[0150] Based on the above premise, suppose there is a new image that needs classification prediction. The output result is category 3 with an accuracy of 80%.

[0151] The visual sensor may adopt conventional Sony IMX477 CMOS image sensor, OmniVision OV5670 CMOS image sensor, ON Semiconductor AR0144 CMOS image sensor, Samsung S5K4H7YX CMOS image sensor, Canon 5D Mark IV CMOS image sensor, etc.

[0152] The sensory system also includes a sensory central anti-electromagnetic interference system and a sensory central storage chip family. The sensory central anti-electromagnetic interference system implements electromagnetic interference (EMI) protection. The sensory central anti-electromagnetic interference system uses EMI shielding materials and covers sensitive components with conductive or magnetic materials to reduce the impact of external electromagnetic waves; the sensory central storage chip family is used to store memory and storage devices for data collected from various sensor networks, including random access memory (RAM): fast-access memory for temporary storage of data in processing and solid-state drive (SSD): used for long-term data storage with fast read and write speeds and high durability.

[0153] The sensory central storage chip family can use conventional Samsung PM9A3 E1.S SSD, Western Digital WD Black SN850 NVMe SSD, SK Hynix Gold P31 NVMe SSD, Crucial P5 Plus NVMe SSD, Intel Optane SSD 905P U.2SSD, etc.

[0154] An inspection hatch 11 is provided on the rear side of the head structure 1, and the inspection hatch 11 is hinged to the head structure 1. Such hinged structure allows the inspection hatch 11 to be opened and stay at a certain angle, so as to facilitate access to the internal structure of the head structure 1. The inspection hatch 11 facilitates replacement and maintenance of the robot's sensory system. The design of the inspection hatch 11 enables technicians to easily replace or upgrade these components, as well as perform routine maintenance and fault diagnosis.

[0155] A memory chip card slot and a hatch 12 are also provided on the rear side of the skull structure 1. The hatch 12 can be detachably installed in the memory chip card slot, and the sensory central memory chip family is installed in the memory chip card slot.

[0156] The skull structure is provided with two groups of visual mounting holes which are symmetrical on the left and right, and the visual system 3 is provided with two groups, and the two groups of visual systems 3 are respectively arranged at the positions of the two groups of visual mounting holes, and the two groups of visual systems 3 have the same structure. In this embodiment, the structure of one group of visual systems 3 is described, and the visual system 3 includes an eye shell 30 and an eyebrow structure 31, an eyelid structure 32 and an eyeball 33 which are installed in the eye shell 3, the eyebrow structure 31 is arranged above the visual mounting hole, the eyelid structure 32 and the eyeball 33 are installed in the visual mounting hole, the eyebrow structure 31 includes an eyebrow movable bracket 311 and two groups of eyebrow movable components 312, and the two groups of eyebrow The movable components 312 are respectively connected to the eyebrow movable brackets 311, and the up and down movement of the eyebrow movable brackets 311 is realized through the eyebrow movable components 312. The eyelid structure 32 includes an upper eyelid bracket 321, an upper eyelid 322, a lower eyelid 323, a lower eyelid bracket 324 and an eyelid movable component. The eyelid movable component controls the opening and closing movement of the upper eyelid bracket 321 and the lower eyelid bracket 324. The eyeball 33 includes a main body assembly 331 and an eye movable component 332. The main body assembly 33 is movably installed in the middle of the eyelid structure 32. The eye movable component 332 controls the up and down movement and rotation of the main body assembly 331.

[0157] The eye housing 30 serves as an outer structure that provides protection and support for the internal components of the eye.

[0158] The eyebrow movable support 31, the upper eyelid support 321, and the lower eyelid support 324 are made of ABS material; the upper eyelid 322 and the lower eyelid 323 are made of high-temperature resistant silicone polymer, on which eyelashes can be installed.

[0159] The eyebrow movable component 312 includes a servo actuator 3121 and an eyebrow connecting rod 3122. The upper end of the eyebrow connecting rod 3122 is connected to an upper movable joint 3123, and the lower end of the eyebrow connecting rod 3122 is connected to a lower movable joint 3124. The servo actuator 3121 is installed on the skull structure. The upper movable joint 3123 is connected to the servo actuator 3121 through an eyebrow movable pin 3125, and can achieve a rotation similar to a universal ball. The eyebrow movable bracket 311 is provided with an eyebrow fixing pin, and the lower movable joint 3124 is connected to the eyebrow fixing pin.

[0160] The servo motor 3121 starts to control the rotation of the eyebrow movable pin 3125. When the eyebrow movable pin 3125 rotates from the bottom to the top, it drives the eyebrow connecting rod 3122 to move upward, and then drives the eyebrow movable bracket 311 to move upward. When the eyebrow movable pin 3125 rotates from the top to the bottom, it drives the eyebrow connecting rod 3122 to move downward, and then drives the eyebrow movable bracket 311 to move downward, thereby controlling the up and down movement of the eyebrow movable bracket 311.

[0161] The two sets of eyebrow movable supports 311 of the two sets of visual systems 3 can perform synchronous or asynchronous movement through the drive of the eyebrow movable component 312, thereby realizing the expression of various emotions by the robot.

[0162] In this embodiment, the eyebrow movable support 311 is controlled by two groups of eyebrow movable components 312 to perform corresponding movements. The eyebrow movable component 312 is connected to the execution unit. After receiving the eyebrow movable support activity trigger instruction, the execution unit sends an eyebrow movable support control signal to the eyebrow movable component 312, thereby controlling the movement of the eyebrow movable support 311.

[0163] The eyebrow movable support activity triggering instruction is the eyebrow movable support up and down movement triggering instruction and the asynchronous movement instruction of two groups of eyebrow movable support activities at the left and right positions, including:

[0164] Visual recognition instructions: When the user's eye expression or gesture is detected by a camera or other visual sensor, the up and down movement of the right eyebrow is triggered according to the user's eye direction or gesture. For example, when the user raises his eyebrows upward or frowns downward, the corresponding movement of the right eyebrow is triggered;

[0165] Voice interaction commands: When the humanoid robot has a voice conversation with the user, the up and down movement of the right eyebrow is triggered according to the content and directionality of the voice command to express the robot's attention or response action.

[0166] The asynchronous motion triggering instructions for the two sets of eyebrow movable bracket activities include

[0167] Visual tracking instructions: Track the user's head movement or facial expression through a camera or other visual sensors. When the user's head or eyes turn left or right, the right eyebrow will be triggered to move slightly left or right to simulate the person's eye gaze or expression changes.

[0168] Voice interaction commands: When the humanoid robot has a voice conversation with the user, the right eyebrow will be triggered to move slightly left and right according to the content and direction of the voice command to express the robot's attention or response action.

[0169] The upper eyelid 322 is fixed on the upper eyelid support 321, and the lower eyelid 323 is fixed on the lower eyelid support 324. The upper eyelid support 321 and the lower eyelid support 324 are respectively provided with tooth ends at the left and right ends. The two tooth ends are coaxial and meshingly arranged inside and outside. When one of the tooth ends is driven to rotate, the upper eyelid support 321 and the lower eyelid support 324 realize opposite rotation due to their meshing movement, which can make the upper eyelid support 321 and the lower eyelid support 324 perform opening and closing movements to realize blinking operations, and the opening and closing frequency can be set according to needs.

[0170] The eyelid movable component provides power for the eyelid movable component. Specifically, the eyelid movable component includes an eyelid opening and closing rotating shaft 325 and a servo servo. The servo servo is installed on the skull structure. The output shaft of the servo servo is fixedly connected to the eyelid opening and closing rotating shaft 325. The eyelid opening and closing rotating shaft 325 is fixedly connected to the tooth end located on the inner side. When the servo servo is started, power is transmitted to the inner tooth end through the eyelid opening and closing rotating shaft 325, thereby realizing the opening and closing movement of the upper eyelid bracket 321 and the lower eyelid bracket 324.

[0171] The two sets of eyelid structures 32 of the two sets of visual systems 3 can perform synchronous opening and closing movements or asynchronous opening and closing movements through the driving of the eyelid movable components.

[0172] In this embodiment, the upper eyelid support 321 and the lower eyelid support 324 are controlled by the eyelid movable component to perform corresponding movements. The servo actuator is connected to the execution unit. After receiving the eyelid support trigger instruction, the execution unit sends the eyelid support control signal to the servo actuator, thereby controlling the movement of the eyelid support.

[0173] The eyelid support triggering instruction includes an instruction for triggering the opening and closing movement of the upper eyelid support 321 and the lower eyelid support 324;

[0174] The upper eyelid support 321 and the lower eyelid support 324 opening and closing movement triggering instructions include

[0175] Visual recognition command: When the user's eye expression or gesture is detected by a camera or other visual sensor, the opening and closing movement of the right upper eyelid bracket is triggered according to the user's eye state. For example, when the user closes or opens his eyes, the corresponding action of the right upper eyelid bracket is triggered;

[0176] Emotion simulation instructions: Analyze emotional signals in the environment based on the emotion recognition algorithm. For example, when the user's tiredness, surprise or joy is detected, the opening and closing movement of the right upper eyelid bracket is triggered to simulate the corresponding changes in eye expression.

[0177] The main body assembly 331 includes an eyeball-shaped shell 3315, an eyeball-shaped shell back cover 3316 and a back cover wire collection slot locking nut 3317. A visual sensor 3311, a variable-focus movable motor seat 3312, an integrated circuit board 3313 and a rubber ring 3314 for fixing a connecting rod seat are fixed in sequence inside the eyeball-shaped shell 3315. The eyeball-shaped shell back cover 3316 is fixed to the end of the eyeball-shaped shell 3315 and is locked by the back cover wire collection slot locking nut 3317.

[0178] The visual sensor 3311 can use a miniature high-definition camera with functions such as automatic exposure / auto focus / auto white balance / auto fill light to capture visual information of the surrounding environment.

[0179] The visual sensor 3311 is part of the visual video tracking system and transmits the captured high-definition video and images to the graphics processing unit for processing.

[0180] The eye movement component 332 is used to control the movement of the eyeball so that it can rotate in the horizontal and vertical directions, thereby changing the direction of the line of sight. The eye movement component 332 includes an eyeball support rod 3320, an eyeball connecting rod 3322 and a servo actuator 3325. The eyeball support rod 3320 is located in the middle and connects the eyeball 33 and the eyeball connecting block 3324. The eyeball connecting block 3324 is installed on the skull structure. The servo actuator 3325 is installed on the eyeball connecting block 3324. Both ends of the eyeball connecting rod 3322 are provided with movable joints. The movable joint at one end is connected to the eyeball contour shell 3315 through a fixing pin, and the other end is connected to the servo actuator 3325 through a movable pin. A group of servo actuators 3325 controls the movement of a group of eyeball connecting rods 3322.

[0181] When a number of servo actuators 3325 control the main body assembly 331 to move forward and the remaining number of servo actuators 3325 control the main body assembly 331 to move backward, the main body assembly 331 can be turned up or down. This embodiment controls different servo actuators 3325 to drive different eyeball connecting rods 3322 to achieve horizontal and vertical rotation of the eyeball, thereby changing the direction of the line of sight.

[0182] In this embodiment, when the robot detects that a person or object has entered its surroundings, the visual video tracking system will begin to capture images and transmit them to the perception center for analysis. If the perception center determines that the line of sight needs to be adjusted to track the target, it sends a corresponding control instruction to the eye movement component 332 to trigger the eye movement;

[0183] The triggering of eye movement mainly includes target detection, target importance assessment, target movement analysis, and the priority and requirements of the current task, which specifically includes the following steps:

[0184] Step A1. Target detection and confirmation:

[0185] 1.1. Detection: The visual video tracking system detects people or objects in the image;

[0186] 1.2. Confirmation: Confirm whether the detected object meets the tracking conditions and characteristics, such as a specific shape, color, or known markers.

[0187] Step A2. Goal importance and priority assessment:

[0188] 2.1. Importance: Assessing the importance of a target depends on the specific requirements of the task (e.g., security monitoring, interactive tasks, or important objects in a specific scenario).

[0189] 2.2 Priority: In a multi-target environment, decide which targets have a higher tracking priority based on the target’s dynamic behavior, mission relevance, or preset rules.

[0190] Step A3. Target dynamic analysis:

[0191] 3.1 Motion tracking: Analyze the target's motion trajectory to determine whether it is moving, its speed and direction.

[0192] 3.2 Predict future position: Use the motion estimation model to predict the target’s future position to determine how to adjust the line of sight most effectively.

[0193] Step A4. Task and environment requirements:

[0194] 4.1 Task requirements: Decide whether to adjust the line of sight based on the robot’s current task requirements (e.g., whether it needs to continuously monitor an area or object) or environmental changes (e.g., other moving objects, light changes, etc.).

[0195] Step A5. Sight adjustment strategy:

[0196] 5.1 Instant adjustment: If the position or speed of the target does not match the predetermined tracking strategy, immediately adjust the sight direction to keep the target in the center of the field of view or in an appropriate position.

[0197] 5.2 Predictive adjustment: Based on the prediction model of target movement, adjust the line of sight direction in advance to reduce reaction delay and improve tracking continuity and accuracy.

[0198] The trigger instructions may include

[0199] Visual tracking instructions: The user's head movement or facial expression is captured in real time through a camera or other visual sensors, and the left and right eyeballs are triggered to move synchronously according to the user's head movement. When the user's head is raised up, lowered, turned left or right, the left and right eyeballs are triggered to move accordingly to simulate the changes in a person's line of sight.

[0200] Gesture recognition commands: The humanoid robot is equipped with gesture recognition technology that triggers eye movements based on the user's hand movements. For example, when the user's finger points in a specific direction, the eye moves in the corresponding direction.

[0201] Context-aware instructions: Through environmental perception technologies such as sound sensors or depth cameras, the sound or object position in the surrounding environment is sensed, thereby triggering the asynchronous movement of the left and right eyeballs. For example, when a humanoid robot detects that the sound comes from the left, the left eyeball may turn to the left, while the right eyeball remains still to simulate the direction of human attention.

[0202] Emotional simulation instructions: Analyze the user's emotional changes based on the emotion recognition algorithm and trigger asynchronous eye movement based on the emotional changes. For example, when the user is detected to be anxious or surprised, the left and right eyeballs may show asynchronous up and down and left and right movements to express the robot's response to the user's emotions.

[0203] When the robot needs to change the direction of its sight, the perception center will analyze the current environment and determine the target position that needs to be adjusted. Specifically, based on the target position and the current position of the eyeball, the execution unit will send a corresponding control signal to the eye movement component 332 to adjust the direction and position of the eyeball to align with the target.

[0204] The direction in which the robot needs to change its line of sight is related to target tracking, task requirements, and environmental changes.

[0205] Target tracking: Deviation of the target position detected by the vision sensor from the expected or planned path;

[0206] Task requirements: For example, the user needs to change their line of sight to adapt to changes in the environment or focus when navigating, avoiding obstacles, or interacting with people.

[0207] Environmental changes: Changes in the environment may require the robot to adjust its line of sight to gain more information or better understand its surroundings.

[0208] The specific steps include:

[0209] Step B1: Determination of target location;

[0210] The specific location or coordinates of the target are usually determined by the following steps:

[0211] 1.1. Image Capture: First, the visual sensor captures the image of the current field of view;

[0212] 1.2. Image processing: Identify objects in the image through the signal processing unit;

[0213] 1.3. Object positioning: Locate the position of an object in three-dimensional space based on a deep learning model;

[0214] 1.4. Coordinate transformation: According to the coordinates of the robot's own position and orientation, the coordinates of the target object are transformed into the position in the global coordinate system;

[0215] Step B2: Adjust the eye position and sight direction;

[0216] The adjustment process includes:

[0217] 2.1. Error calculation: Calculate the deviation between the target position and the current line of sight center;

[0218] 2.2. Motion planning: Calculate the angle and direction of eye movement based on the deviation;

[0219] 2.3. Perform adjustment: By controlling the eye movement component 332, adjust the line of sight so that the target is located in the center of the visual field or other predetermined position.

[0220] The voice instructions and voice interaction instructions in this embodiment are captured by the auditory sensor, processed by the signal processing unit, and issued by the perception center to the execution unit; the visual tracking / recognition instructions and emotion simulation instructions are captured by the visual sensor, processed by the signal processing unit, and issued by the perception center to the execution unit; and the emotion expression instructions are issued by the perception center to the execution unit.

[0221] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

Claims

1. The eye structure of the bionic service humanoid robot is installed on the head structure, and is characterized by: It includes a sensory system and a visual system. The sensory system includes a perception center, a sensor network, a signal processing unit and an execution unit. The sensor network is connected to the signal processing unit, the signal processing unit is connected to the perception center, the perception center is connected to the execution unit, and the execution unit is connected to the visual system. The sensor network includes several visual sensors that capture visual information of the external environment; The signal processing unit is used to receive the original signal from the sensor network, perform signal processing and preprocessing, and transmit the processed signal to the perception center; The perception center receives the processed signals, performs analysis and decision-making, and obtains the status of the external environment and the required response; The execution unit receives the triggering instruction from the perception center and executes the corresponding action or task; The visual system receives the control signal sent by the execution unit and performs corresponding movements.

2. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The signal processing unit includes a graphics processing unit. The graphics processing unit and the visual sensor constitute a visual video tracking system. The visual sensor captures visual information of the surrounding environment and transmits it to the graphics processing unit for analysis, recognition and tracking. The graphics processing unit has a built-in image processing algorithm. The visual sensor captures continuous image frames and recognizes and extracts the required image or object through the image processing algorithm. Specifically, the following steps are included: Step S1: The visual sensor captures real-time images or videos in the environment and transmits them to the graphics processing unit; Step S2: The image processing algorithm pre-processes the captured image; Step S3: Feature extraction: extracting specific information from the preprocessed image; The feature extraction methods include edge detection based on edge detection algorithm, corner detection based on corner detection algorithm, and texture analysis based on texture analysis algorithm; Step S4: object recognition; Identify and classify objects in images using a given machine learning model; Step S5: object positioning and tracking; After object recognition, the specific location of the object in the image is determined based on the bounding box, which frames each recognized object in the image to locate it; Step S6: output the result; The graphics processing unit outputs the results of recognition and positioning to the perception center, which analyzes the results to determine whether to generate a corresponding decision. If so, it adjusts the parameters and executes step S2 to process the newly captured image. If not, it adjusts the model, calls the deep learning model, and executes step S4.

3. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The sensory system also includes a sensory central anti-electromagnetic wave interference system and a sensory central storage chip family. The sensory central anti-electromagnetic wave interference system realizes electromagnetic interference protection. The sensory central anti-electromagnetic wave interference system uses conductive or magnetic materials to cover sensitive components. The sensory central storage chip family is used to store memory and storage devices for storing data collected from the sensor network, including random access memory, fast access memory and solid state hard disk. The fast access memory is used to temporarily store data in processing. Solid-state drives are used for long-term data storage.

4. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: The visual system includes an eye shell and an eyebrow structure, an eyelid structure and an eyeball installed in the eye shell. The eyebrow structure includes an eyebrow movable support and two groups of eyebrow movable components. The two groups of eyebrow movable components are respectively connected to the eyebrow movable support. The eyelid structure includes an upper eyelid support, an upper eyelid, a lower eyelid, a lower eyelid support and an eyelid movable component. The eyelid movable component controls the opening and closing movements of the upper eyelid support and the lower eyelid support. The eyeball includes a body assembly and an eye movable component. The body assembly is movably installed in the middle of the eyelid structure. The eye movable component controls the up and down movement and rotation of the body assembly.

5. The eye structure of the bionic service humanoid robot according to claim 4, characterized in that: The eyebrow movable component includes a servo servo motor and an eyebrow connecting rod, the upper end of the eyebrow connecting rod is connected to an upper movable joint, the lower end of the eyebrow connecting rod is connected to a lower movable joint, the servo servo motor is installed on the skull structure, the upper movable joint is connected to the servo servo motor through an eyebrow movable pin, the eyebrow movable bracket is provided with an eyebrow fixing pin, and the lower movable joint is connected to the eyebrow fixing pin.

6. The eye structure of the bionic service humanoid robot according to claim 4, characterized in that: The eyebrow movable support is controlled by two groups of eyebrow movable components to perform corresponding movements. The eyebrow movable components are connected to the execution unit. After receiving the eyebrow movable support activity trigger instruction and processing it, the execution unit sends an eyebrow movable support control signal to the eyebrow movable component to control the eyebrow movable support structure to move synchronously or asynchronously.

7. The eye structure of the bionic service humanoid robot according to claim 4, characterized in that: The main body assembly includes an eyeball-shaped shell, a rear cover of the eyeball-shaped shell and a rear cover wire collection slot locking nut. A visual sensor, a variable-focus movable motor seat, an integrated circuit board and a rubber ring for fixing a connecting rod seat are fixed in sequence inside the eyeball-shaped shell. The rear cover of the eyeball-shaped shell is fixed to the end of the eyeball-shaped shell and is locked by the rear cover wire collection slot locking nut.

8. The eye structure of the bionic service humanoid robot according to claim 2, characterized in that: When the robot detects that a person or object has entered its surroundings, the visual video tracking system captures the image and transmits it to the perception center for analysis. If the perception center determines that the line of sight needs to be adjusted to track the target, it sends a corresponding control instruction to the eye movement component 332 to trigger the eye movement: The triggering of eye movements includes target detection, target importance assessment, analysis of target movement, and the priority and requirements of the current task, which specifically includes the following steps: Step A1. Target detection and confirmation: 1.

1. Detection: The visual video tracking system detects people or objects in the image; 1.

2. Confirmation: Confirm whether the detected object meets the tracking conditions and characteristics; Step A2. Goal importance and priority assessment: 2.

1. Importance: Evaluate the importance of the goal, which depends on the specific requirements of the task; 2.2 Priority: In a multi-target environment, determine which targets have a higher tracking priority based on the target's dynamic behavior, task relevance, or preset rules; Step A3. Target dynamic analysis: 3.1 Motion tracking: Analyze the target's motion trajectory to determine whether it is moving, its speed and direction; 3.2 Predicting future position: Use the motion estimation model to predict the target’s future position; Step A4. Task and environment requirements: 4.1 Task requirements: Decide whether to adjust the line of sight based on the robot’s current task requirements or environmental changes; Step A5. Sight adjustment strategy: 5.1 Instant adjustment: If the position or speed of the target does not match the predetermined tracking strategy, immediately adjust the sight direction to keep the target in the center of the field of view or in an appropriate position; 5.2 Predictive adjustment: Adjust the sight direction in advance based on the prediction model of target motion.

9. The eye structure of the bionic service humanoid robot according to claim 1, characterized in that: When the robot needs to change the direction of sight, the perception center analyzes the current environment, determines the target position that needs to be adjusted according to the target position and the current position of the eyeball, and the execution unit sends a corresponding control signal to the eye movement component 332 to adjust the direction and position of the eyeball, which specifically includes the following steps: Step B1: Determination of target location; The specific location or coordinates of the target are usually determined by the following steps: 1.

1. Image Capture: First, the visual sensor captures the image of the current field of view; 1.

2. Image processing: Identify objects in the image through the signal processing unit; 1.

3. Object positioning: Locate the position of an object in three-dimensional space based on a deep learning model; 1.

4. Coordinate transformation: According to the coordinates of the robot's own position and orientation, the coordinates of the target object are transformed into the position in the global coordinate system; Step B2: Adjust the eye position and sight direction; The adjustment process includes: 2.

1. Error calculation: Calculate the deviation between the target position and the current line of sight center; 2.

2. Motion planning: Calculate the angle and direction of eye movement based on the deviation; 2.

3. Perform adjustment: By controlling the eye movement component 332, adjust the line of sight so that the target is located in the center of the visual field or other predetermined position.

Images