Cognitive development method and system based on bimodal semi-supervised learning network

Abstract

The invention discloses a cognitive development method and system based on a bimodal semi-supervised learning network, and the method comprises the steps: receiving a voice signal, obtaining the meta-information in a statement, and transmitting the meta-information to a SOINN network; the SOINN network receives a visual signal and extracts a color feature and a shape feature of the visual signal; clustering by adopting a density peak clustering algorithm according to the meta-information, the color features and the shape features to obtain a clustering result; carrying out feature association according to a clustering result in combination with the color features and the shape features to obtain an association result; and performing callback operation according to the association result. New knowledge can be learned without forgetting previous knowledge; and the learning network can simultaneously meet the conditions with labels and without labels.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a cognitive development method and system based on a dual-modal semi-supervised learning network. Background Art

[0002] Against the backdrop of the rapid development of intelligent robotics technology, cognitive development has become a core research focus in this field. Given the trend of intelligent robots increasingly integrating into people's daily lives, they need to demonstrate efficient recognition and understanding capabilities in the process of interacting with humans. Traditionally, the knowledge representation within robots mostly relies on human presets and programming, which is inadequate when faced with unknown and dynamically changing environments. To address this limitation, robots should have the ability to self-learn, adapt, and evolve their cognitive systems during the interaction process.

[0003] There are many difficulties in overcoming this limitation. For example, robots cannot automatically associate their own observations with the names they know like humans do; and when encountering new states, the state list cannot be expanded online. Therefore, autonomous cognitive development is still a huge challenge for robots. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a cognitive development method and system based on a bimodal semi-supervised learning network that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of the present invention, a cognitive development method based on a bimodal semi-supervised learning network is provided, the cognitive development method comprising:

[0006] Receive a speech signal, obtain meta information in the sentence, and send the meta information to the SOINN network;

[0007] The SOINN network receives a visual signal and extracts color features and shape features of the visual signal;

[0008] Performing clustering using a density peak clustering algorithm according to the meta information, the color feature, and the shape feature to obtain a clustering result;

[0009] Performing feature association based on the clustering result and in combination with the color feature and the shape feature to obtain an association result;

[0010] A callback operation is performed according to the association result.

[0011] Optionally, the receiving a voice signal and acquiring meta information in a sentence specifically includes:

[0012] Extract and entity e separately i or ej There is a dependent node e' with a COO parallel structure or an ATT definite structure relationship i and e' j , output a dependency node of the entity e; where e i and e j Respectively represent the i-th and j-th entities, e' i and e j 'represent the i-th and j-th dependency nodes respectively;

[0013] The entity e is the dependency syntactic analysis and part-of-speech tagging result of the sentence in which it is located;

[0014] Extract the nearest verb that has a dependency relationship with the dependency-related node of the second entity;

[0015] Get the nearest verb that has SBV or FOB relationship with the dependent association node of the first entity;

[0016] By judging whether the second entity verb and the first entity verb are the same verb or a COO parallel structure relationship, the entity pair is determined. <e i , e j >'s nearest syntactically dependent verb;

[0017] Shape labels, color labels and object names are selected from the nearest syntactically dependent verbs as meta-information.

[0018] Optionally, the SOINN network receives a visual signal, and extracting the color feature and shape feature of the visual signal specifically includes:

[0019] For an RGB image, the image is divided into four sub-blocks, the color histogram of the three channels of R, G, and B of each sub-block is extracted, and the color features of the four sub-blocks are spliced ​​to obtain the color features;

[0020] Based on the Canny edge detection algorithm, the edge points in the image are identified. Based on the edge image, the findContours function in OpenCV is used to extract the contour of the image.

[0021] The extracted contour is represented parameterically and shape features are extracted from the parameterized representation of the contour.

[0022] Optionally, the shape features include length, area, convexity, curvature and direction of the contour.

[0023] Optionally, performing clustering by using a density peak clustering algorithm according to the meta information, the color feature, and the shape feature to obtain a clustering result specifically includes:

[0024] For all samples D in the meta-information, calculate the Euclidean distance between any two samples and establish the distance matrix D m ;

[0025] Use the iterative method to select the center point and obtain the center point set C of D;

[0026] Perform local clustering with each center point in C as the cluster center to obtain the cluster set C on D LS ={C L1 ,C L2 ,...,C LP}: Among them, C Li represents the i-th cluster in the cluster set;

[0027] For the cluster set C on D LS For each cluster in (1≤i≤P), use the center point of the label pair of the labeled samples in to label and obtain the label set of the center point set C;

[0028] Construct a graph G with each sample in the center point set C as a vertex, calculate the similarity between any two vertices in G, establish a similarity matrix W, and use Y c Complete the sample labeling process based on LGC theory and obtain the predicted label set Y of the center point set C p ; Among them, Y c The label set representing the center point set C;

[0029] The similarity between any two vertices in G is calculated by the following formula:

[0030]

[0031] Among them, x i and x j Represent the i-th and j-th vertices respectively, and σ represents the distance between the two vertices;

[0032] Take advantage of Y p The labels of the center points in the cluster are used to label all unlabeled samples in the cluster where each center point is located.

[0033] Optionally, the performing feature association according to the clustering result and in combination with the color feature and the shape feature specifically includes:

[0034] Clustering results by shape, color and name (c i ,s j ,n k ), nodes whose categories change during shape and color learning And all the category sets S and C in the shape and color networks are used as input for feature association; where c irepresents the i-th clustering result of shape features, s j represents the jth clustering result of the color feature, n k represents the kth clustering result of the name feature, and They represent the i-th node before and after the category change in the shape learning process, respectively. and They represent the j-th node before and after the category change in the color learning process, respectively.

[0035] Optionally, the feature association specifically includes:

[0036] Find the same as (c i ,s j ,n k ) The same node, if the node exists, the activation count increases by 1, M i (t) = 1, update f i (z) and v i (z); If the node does not exist, create a new node M i (t) = 1, f r (0) = f(0), v r (0) = v (0); where M i (t) represents the cumulative activation value;

[0037] If the node's concept has a category that does not belong to the shape category set S or the color category set C, delete the node;

[0038] turn up The corresponding nodes, and use Replace Generate new concepts. If the newly generated concepts do not exist, create new nodes.

[0039] turn up The corresponding nodes, and use Replace Generate new concepts. If the newly generated concepts do not exist, create new nodes.

[0040] If there is a node M i (t)<M min , delete node, M i (t)>M max , the node M i (t) = 1, forming a permanent memory; where M i (t) represents the cumulative activation value, M min and M max Represent the minimum and maximum values ​​of the cumulative activation values ​​respectively.

[0041] Optionally, the performing a callback operation according to the association result specifically includes:

[0042] When the network receives auditory information, it compares the speech information in the shape, color and name semantic libraries to find the corresponding node; if the speech is the semantics of shape or color, it finds the node with the same label and the most activation times in the corresponding network and returns the weight;

[0043] If the voice is name n i , find n in the associated network i The activated node N determines the category of shape and color in N and outputs the weight of the corresponding node in the learning network;

[0044] When the network receives visual information, it finds the best matching nodes in the shape and color processing networks, and outputs the categories i and j corresponding to the nodes; then the associated feature network uses s i and c j Represent these two results and find (s i ,c j )Activated nodes;

[0045] If node N is activated, the name information in N is the output of this callback;

[0046] If no node is activated, perform network follow-up processing.

[0047] Optionally, the network subsequent processing specifically includes:

[0048] The associated feature network looks up the shape s i and color c j All nodes N activated l1,l2,... ;

[0049] Get the shape and color class values ​​m, n in node N, N∈N l,k ;

[0050] Find the node in class m in the shape learning network with the smallest distance to the current input, and record the distance dist1;

[0051] Find the node in class n in the color learning network with the smallest distance to the current input, and record the distance dist2;

[0052] dist=dist1+dist2

[0053] If N l1,l2,... If there are unprocessed nodes, the shape and color class values ​​of node N are repeatedly obtained;

[0054] Find the node N with the smallest dist and output its name information.

[0055] The present invention also provides a cognitive development system based on a dual-modal semi-supervised learning network, the cognitive development system comprising:

[0056] A voice signal receiving module, used to receive a voice signal, obtain meta information in a sentence, and send the meta information to the SOINN network;

[0057] A visual signal receiving module, used for the SOINN network to receive visual signals and extract color features and shape features of the visual signals;

[0058] A density peak clustering algorithm module, used to perform clustering using a density peak clustering algorithm according to the meta information, the color feature and the shape feature to obtain a clustering result;

[0059] A feature association module, used to perform feature association based on the clustering result and in combination with the color feature and the shape feature to obtain an association result;

[0060] The callback operation module is used to perform a callback operation according to the association result.

[0061] The present invention provides a cognitive development method and system based on a bimodal semi-supervised learning network, the cognitive development method comprising: receiving a speech signal, obtaining meta-information in a sentence, and sending the meta-information to a SOINN network; the SOINN network receives a visual signal, extracts the color features and shape features of the visual signal; clusters the meta-information, the color features, and the shape features using a density peak clustering algorithm to obtain a clustering result; performs feature association based on the clustering result and in combination with the color features and the shape features to obtain an association result; and performs a callback operation based on the association result. New knowledge can be learned without forgetting previous knowledge; the learning network used can simultaneously meet both labeled and unlabeled situations.

[0062] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0064] Figure 1A flowchart of a cognitive development method based on a bimodal semi-supervised learning network is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0067] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0068] Step 1: Voice Processing

[0069] Convert the input speech into sentences and obtain the meta-information in the sentences based on dependency syntactic analysis and part of speech. Specifically:

[0070] Step 1-1, extract entity e separately i or e j There is a dependent node e' with a COO parallel structure or an ATT definite structure relationship i and e' j , that is: by inputting an entity e (the dependency syntactic analysis and part-of-speech tagging results of the sentence where the entity is located), a dependency association node of the entity e is output; wherein, e i and e j Respectively represent the i-th and j-th entities, e' i and e' j Represent the i-th and j-th dependency nodes respectively;

[0071] Step 1-2, extracting the nearest verb that has a dependency relationship with the dependency-associated node of the second entity;

[0072] Step 1-3, obtain the nearest verb that has a SBV or FOB relationship with the dependency-related node of the first entity;

[0073] Step 1-4: Determine whether the second entity verb and the first entity verb are the same verb or a COO parallel structure relationship, and then determine the entity pair. <e i , e j >'s nearest syntactic dependent verb.

[0074] Shape labels, color labels and object names are selected from the nearest syntactically dependent verbs as meta-information and fed into the SOINN network.

[0075] Step 2: Visual Learning

[0076] At the same time, the SOINN network receives visual signals, and the SOINN network is provided with two network channels, which are respectively used to learn and extract the color features and shape features of the object.

[0077] For an RGB image, the image is divided into four sub-blocks, the color histogram of the three channels R, G, and B of each sub-block is extracted, and then the color features of the four sub-blocks are spliced ​​to obtain the color features.

[0078] Based on the Canny edge detection algorithm, edge points in the image (locations where brightness changes significantly in the image) are identified. Based on the edge image, the findContours function in OpenCV is used to extract the contour of the image. The extracted contour is parameterized and shape features are extracted from the parameterized representation of the contour.

[0079] Shape features include the length, area, convexity, curvature, and direction of the contour.

[0080] After extracting the color features and shape features from the image, the density peak clustering algorithm is executed for clustering.

[0081] (1) For all samples D in the metadata, calculate the Euclidean distance between any two samples and establish a distance matrix.

[0082] (2) Use the iterative method to select the center point and obtain the center point set C of D, that is:

[0083] For each sample x in sample D i (1≤i≤n), n is the number of samples in D. The local density ρ is calculated according to the traditional DPC algorithm. i With distance value δ i , and calculate γ i =ρ i *δ i Sort all samples in D by γ value from large to small, and add the sorted sample numbers to the array q in order, that is, γ q[1] ≥γ q[2] ≥......≥γ q[n] ; To filter the center points according to the order of appearance of each sample in array q.

[0084] (3) Perform local clustering with each center point in C as the cluster center to obtain the cluster set C on D LS ={C L1 ,CL2 ,...,C LP}, where C Li Represents the i-th cluster in the cluster set; specifically:

[0085] The clustering array qc is introduced to record the subscripts of the samples in sample D that are closest to the current sample and have higher local density. i , qc[i] represents the number of cells in D that are separated from x i The subscript of the nearest sample with higher local density, if there is no sample in D with a higher local density than x i For samples with higher density, qc[i] stores x i The subscript of .

[0086] When clustering, if x qc[i] As the center point, then point x i Classification x qc[i] Otherwise, point x i Iterative classification to x qc[i] The closest and more dense sample x qc[q'c[i]] , and so on, until x i Until it belongs to the cluster to which a center point belongs.

[0087] (4) For C LS Each cluster C in Li (1≤i≤P), using C Li The label pairs of labeled samples in C Li Mark the center points of the center point set C to obtain the label set Y c .

[0088] (5) Construct a graph G with each sample in the center point set C as a vertex, calculate the similarity between any two vertices in G, establish a similarity matrix W, and then use Y c Complete the sample labeling process based on LGC theory and obtain the predicted label set Y of the center point set C p ; Among them, Y c Represents the label set of the center point set C.

[0089] The similarity between any two vertices in G is calculated by the following formula:

[0090]

[0091] Among them, x i and x j Represent the i-th and j-th vertices respectively, and σ represents the distance between the two vertices;

[0092] (6)Use Y p The labels of the center points in the cluster are used to label all unlabeled samples in the cluster where each center point is located.

[0093] Step 3: Feature association

[0094] Clustering results by shape, color and name (c i ,s j ,n k ), nodes whose categories change during shape and color learning And all the category sets S and C in the shape and color networks are used as input for feature association; where c i represents the i-th clustering result of shape features, s j represents the jth clustering result of the color feature, n k represents the kth clustering result of the name feature, and They represent the i-th node before and after the category change in the shape learning process, respectively. and They represent the jth node before and after the category change in the color learning process, specifically:

[0095] (1) Find the i ,s j ,n k ) The same node, if the node exists, the activation count increases by 1, M i (t) = 1, update f i (z) and v i (z); If the node does not exist, create a new node M i (t) = 1, f r (0) = f(0), v r (0) = v (0); where M i (t) represents the cumulative activation value;

[0096] (2) If the node’s concept contains a category that does not belong to the shape category set S or the color category set C, delete the node.

[0097] (3) Find The corresponding nodes, and use Replace Generate a new concept. If the newly generated concept does not exist, create a new node.

[0098] (4) Find The corresponding nodes, and use Replace Generate a new concept. If the newly generated concept does not exist, create a new node.

[0099] (5) If there is a node M i (t)<Mmin , delete node, M i (t)>M max , the node M i (t) = 1, forming a permanent memory; where M i (t) represents the cumulative activation value, M min and M max Represent the minimum and maximum values ​​of the cumulative activation values ​​respectively.

[0100] After learning through the above steps, associations are formed between the various inputs, and when the robot sees or hears related content again, it calls back the relevant knowledge.

[0101] When the network receives auditory information, it compares the speech information in the shape, color, and name semantic libraries to find the corresponding node. If the speech is the semantics of shape or color, find the node with the same label and the most activation times in the corresponding network and return its weight. If the speech is the name i , find n in the associated network i The activated node N determines the category of shape and color in N and outputs the weight of the corresponding node in the learning network.

[0102] When the network receives visual information, it finds the best matching nodes in the shape and color processing networks, and outputs the categories i and j corresponding to the nodes; then the associated feature network uses s i and c j Represent these two results and find (s i ,c j ) The node activated by the callback; if node N is activated, the name information in N is the output of this callback.

[0103] If no node is activated, it does not necessarily mean that the current input has not been learned, because in the process of visual learning, similar data may not be completely divided into the same cluster. The subsequent processing of the network is as follows:

[0104] (1) The associated feature network searches for the shape s i and color c j All nodes activated {N l1,l2,...}.

[0105] (2) Get the shape and color class values ​​m, n in node N, N∈N l,k .

[0106] (3) Find the node in class m in the shape learning network that has the smallest distance to the current input and record the distance dist1.

[0107] (4) Find the node in class n in the color learning network with the smallest distance to the current input and record the distance dist2;

[0108] dist=dist1+dist2

[0109] (5) If {N l1,l2,...}, repeat step (2).

[0110] (6) Find the node N with the smallest dist and output its name information.

[0111] Callback is the most important process in the top-down response and the basis of the other two response processes. The learned audio-visual association is fully utilized in this process. Allowing each layer to learn and process specific information and call back layer by layer is closer to the way the brain processes information, and is more concise and effective when associating bimodal information.

[0112] Beneficial effects: (1) New knowledge can be learned without forgetting previous knowledge; (2) The learning network used can satisfy both labeled and unlabeled situations.

[0113] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cognitive development method based on a bimodal semi-supervised learning network, characterized in that: The cognitive development method comprises: Receive a speech signal, obtain meta information in the sentence, and send the meta information to the SOINN network; The SOINN network receives a visual signal and extracts color features and shape features of the visual signal; Performing clustering using a density peak clustering algorithm according to the meta information, the color feature, and the shape feature to obtain a clustering result; Performing feature association based on the clustering result and in combination with the color feature and the shape feature to obtain an association result; A callback operation is performed according to the association result.

2. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 1, characterized in that: The receiving of the speech signal and obtaining the meta-information in the sentence specifically includes: Extract and entity e separately i or e j There is a dependent node e' with a COO parallel structure or an ATT definite structure relationship i and e' j , output a dependency node of the entity e; where e i and e j Respectively represent the i-th and j-th entities, e' i and e' j Represent the i-th and j-th dependency nodes respectively; The entity e is the dependency syntactic analysis and part-of-speech tagging result of the sentence in which it is located; Extract the nearest verb that has a dependency relationship with the dependency-related node of the second entity; Get the nearest verb that has SBV or FOB relationship with the dependent association node of the first entity; By judging whether the second entity verb and the first entity verb are the same verb or a COO parallel structure relationship, the entity pair is determined. <e i , e j >'s nearest syntactic dependent verb; among them, e i and e j Represent the i-th and j-th entities respectively; Shape labels, color labels and object names are selected from the nearest syntactically dependent verbs as meta-information.

3. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 1, characterized in that: The SOINN network receives a visual signal, and extracting the color feature and shape feature of the visual signal specifically includes: For an RGB image, the image is divided into four sub-blocks, the color histogram of the three channels of R, G, and B of each sub-block is extracted, and the color features of the four sub-blocks are spliced ​​to obtain the color features; Based on the Canny edge detection algorithm, the edge points in the image are identified. Based on the edge image, the findContours function in OpenCV is used to extract the contour of the image. The extracted contour is represented parameterically and shape features are extracted from the parameterized representation of the contour.

4. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 3, characterized in that: The shape characteristics include the length, area, convexity, curvature and direction of the contour.

5. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 1, characterized in that: The clustering is performed using a density peak clustering algorithm according to the meta information, the color feature and the shape feature to obtain a clustering result specifically comprising: For all samples D in the meta-information, calculate the Euclidean distance between any two samples and establish the distance matrix D m ; Use the iterative method to select the center point and obtain the center point set C of D; Perform local clustering with each center point in C as the cluster center to obtain the cluster set C on D LS ={C L1 ,C L2 ,...,C LP }: Among them, C Li represents the i-th cluster in the cluster set; For the cluster set C on D LS For each cluster in (1≤i≤P), use the center point of the label pair of the labeled samples in to label and obtain the label set of the center point set C; Construct graph G with each sample in the center point set C as a vertex, calculate the similarity between any two vertices in G, establish similarity matrix W, and use Y c Complete the sample labeling process based on LGC theory and obtain the predicted label set Y of the center point set C p ; Among them, Y c The label set representing the center point set C; The similarity between any two vertices in G is calculated by the following formula: Among them, x i and x j Represent the i-th and j-th vertices respectively, and σ represents the distance between the two vertices; Take advantage of Y p The labels of the center points in the cluster are used to label all unlabeled samples in the cluster where each center point is located.

6. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 1, characterized in that: The performing feature association according to the clustering result and in combination with the color feature and the shape feature specifically includes: Clustering results by shape, color and name (c i ,s j ,n k ), nodes whose categories change during shape and color learning And all the category sets S and C in the shape and color networks are used as input for feature association; where c i represents the i-th clustering result of shape features, s j represents the jth clustering result of the color feature, n k represents the kth clustering result of the name feature, and They represent the i-th node before and after the category change in the shape learning process, respectively. and They represent the j-th node before and after the category change in the color learning process, respectively.

7. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 6, characterized in that: The feature association specifically includes: Find the same as (c i ,s j ,n k ) The same node, if the node exists, the activation count increases by 1, M i (t) = 1, update f i (z) and v i (z); If the node does not exist, create a new node M i (t) = 1, f r (0) = f(0), v r (0) = v (0); where (c i ,s j ,n k ) represents the clustering results of shape, color and name, M i (t) represents the cumulative activation value; If the node's concept has a category that does not belong to the shape category set S or the color category set C, delete the node; turn up The corresponding nodes, and use Replace Generate a new concept. If the newly generated concept does not exist, create a new node; and They represent the i-th node before and after the category change in the shape learning process; turn up The corresponding nodes, and use Replace Generate a new concept. If the newly generated concept does not exist, create a new node; and They represent the jth node before and after the category change in the color learning process; If there is a node M i (t)<M min , delete node, M i (t)>M max , the node M i (t) = 1, forming a permanent memory; where M i (t) represents the cumulative activation value, M min and M max Represent the minimum and maximum values ​​of the cumulative activation values ​​respectively.

8. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 1, characterized in that: The callback operation according to the association result specifically includes: When the network receives auditory information, it compares the speech information in the shape, color and name semantic libraries to find the corresponding node; if the speech is the semantics of shape or color, it finds the node with the same label and the most activation times in the corresponding network and returns the weight; If the voice is name n i , find n in the associated network i The activated node N determines the category of shape and color in N and outputs the weight of the corresponding node in the learning network; When the network receives visual information, it finds the best matching nodes in the shape and color processing networks respectively, and outputs the categories i and j corresponding to the nodes; then the associated feature network uses s i and c j Represent these two results and find (s i ,c j )Activated nodes; If node N is activated, the name information in N is the output of this callback; If no node is activated, perform network follow-up processing.

9. A cognitive development method based on a dual-modal semi-supervised learning network according to claim 8, characterized in that: The network subsequent processing specifically includes: The associated feature network looks up the shape s i and color c j All activated nodes N l1,l2,... ; Get the shape and color class values ​​m, n in node N, N∈N l,k ; Find the node in class m in the shape learning network with the smallest distance to the current input, and record the distance dist1; Find the node in class n in the color learning network with the smallest distance to the current input, and record the distance dist2; dist=dist1+dist2 If N l1,l2,... If there are unprocessed nodes, the shape and color class values ​​of node N are repeatedly obtained; Find the node N with the smallest dist and output its name information.

10. A cognitive development system based on a bimodal semi-supervised learning network, characterized in that: The cognitive development system comprises: A voice signal receiving module, used to receive a voice signal, obtain meta information in a sentence, and send the meta information to the SOINN network; A visual signal receiving module, used for the SOINN network to receive visual signals and extract color features and shape features of the visual signals; A density peak clustering algorithm module, used to perform clustering using a density peak clustering algorithm according to the meta information, the color feature and the shape feature to obtain a clustering result; A feature association module, used to perform feature association based on the clustering result and in combination with the color feature and the shape feature to obtain an association result; The callback operation module is used to perform a callback operation according to the association result.

Images