An immersive-based bilingual semantic relation system for young children

Abstract

The application discloses an immersion-based bilingual semantic association system for infants, relates to the field of natural language processing and semantic analysis, and comprises the following modules: an acquisition module, which acquires the voice, image, real object and action of an infant; a semantic association network construction module, which establishes an individualized cross-language semantic association network according to the co-occurrence times of a mother tongue label and a target language label of the same object in the field; a code switching detection module, which determines the code switching points in the mixed speech of the infant; a bidirectional semantic signal module, which labels the node corresponding to the word spoken in the target language at the code switching points as an activated node and labels the node corresponding to the concept expressed in the mother language as a gap node; and an immersive presentation module, which selects the target language content to be presented according to the distribution of the activated node and the gap node. The application solves the problems that the code switching signal is discarded as noise and it is difficult to establish and drive an individualized cross-language semantic association according to the code switching signal.

Description

Technical Field

[0001] This application relates to the fields of natural language processing and semantic analysis, and in particular to an immersive bilingual semantic association system for young children. Background Technology

[0002] In the fields of natural language processing and semantic analysis, cross-linguistic semantic understanding is used to establish semantic correspondences between words in two languages, serving as a fundamental step in bilingual information processing. In immersive bilingual environments, young children use their native language while simultaneously being exposed to the target language. Their spontaneous speech often exhibits a hybrid form, with individual target language words interspersed within a native language sentence, or the language temporarily switching back to the native language when expressing a concept. Processing this hybrid speech requires simultaneously identifying its language affiliation and semantic orientation, falling under the scope of natural language processing and semantic analysis.

[0003] To establish semantic correspondences between two languages, existing technologies typically rely on pre-built external resources. CN118246461A discloses a cross-lingual semantic understanding scheme that uses a bilingual dictionary to replace words in queries and annotations with corresponding words in other languages, and trains a masked language model on a mixed-language corpus to obtain a multilingual semantic representation. Its cross-lingual correspondences are derived from a general bilingual dictionary and a large-scale pre-trained corpus. CN112101047A addresses the scarcity of parallel corpora by using back-translation to expand the corpus and matching terms with placeholders during the translation process. CN110083840A organizes words based on letter meanings and word root and affix rules to aid in semantic understanding. A common premise of these schemes is that the cross-lingual semantic correspondences are derived from general dictionaries, parallel corpora, or fixed word formation rules; they are group-oriented, pre-determined, and relatively static mappings.

[0004] However, this premise does not always hold true when dealing with individual children. The translations provided by general dictionaries are based on group statistics, while the meaning of the same target word for a particular child is often specific. For example, "car" might only refer to a toy car, and "rice" might correspond to "rice" or "rice" depending on the context. When the pre-determined translation does not match the child's actual meaning, processing based on general resources cannot correct this. More importantly, to obtain clean monolingual input, the above processing often treats language switching in children's mixed speech as noise and removes or normalizes it. For example, if a child utters a target word within a sentence in their native language, this switching is treated as noise and removed, preventing the system from recording that the child can express the concept in the target language. Conversely, if a child switches back to their native language when expressing a concept, this switch is also erased, preventing the system from knowing that the concept is missing in the target language. Thus, the presented content can only proceed according to an externally pre-set vocabulary and fixed order, rather than based on the child's current actual cross-language mastery.

[0005] The aforementioned solutions collectively overlook the following technical problem: the locations where a child switches between their native language and target language in mixed speech are discrete, detectable signals indicating their cross-linguistic semantic mastery, signals that have long been treated as noise that needs to be removed. Specifically, the words spoken in the target language at the switching location point to concepts the child can already express in the target language; the concepts expressed in the native language at the switching location point to concepts the child is still lacking in the target language; simultaneously, the native language discourse surrounding the switching location naturally provides context to dissolve the specific meaning of the target language words. This information is lost along with the switching in existing processing methods and is not used to establish individual-oriented cross-linguistic semantic connections.

[0006] In addition to the above, two unresolved issues remain in the continuous presentation of immersive bilingual scenarios. First, the starting point for forgetting target words varies from person to person and from word to word. Recalling words according to a fixed review order often occurs later or earlier than the moment when children actually begin to forget, lacking early criteria for judging forgetting. Second, young children primarily rely on auditory discrimination and easily confuse cross-linguistic words that are phonetically similar but semantically unrelated. However, phonetic similarity or semantic unrelatedness alone is insufficient to distinguish such misclassification from normal translation.

[0007] Therefore, how to break free from the dependence on general dictionaries and static corpora, establish individual-oriented cross-linguistic semantic connections from children's own real-time discourse, and drive immersive presentation based on this has become an unsolved problem in this field. Summary of the Invention

[0008] The technical problem to be solved by this application is to provide an immersive bilingual semantic association system for young children, which addresses the issue that code-switching signals in young children's mixed speech are discarded as noise and that it is difficult to establish and drive individualized cross-linguistic semantic associations based on them.

[0009] To achieve the above objectives, this application adopts the following technical solution: An immersive bilingual semantic association system for preschoolers includes: a data acquisition module for acquiring the preschooler's speech in an immersive scene, images and objects in the immersive scene, and the preschooler's pointing or grasping actions. The semantic association network construction module is used to establish a pair of related nodes between mother language tags and target language tags whose co-occurrence frequency reaches a set threshold in the individualized cross-lingual semantic association network (hereinafter referred to as semantic association network) based on the number of times the mother language tag and the target language tag co-occur when the same referent is present. The code-switching detection module is used to identify code-switching points between the native language and the target language in children's mixed language discourse; The bidirectional semantic signal module is used to mark the node corresponding to the word spoken in the target language at the code-switching point in the semantic association network as an activated node, and to mark the node corresponding to the concept expressed by the child in the native language at the code-switching point in the target language as a gap node. The immersive presentation module is used to select, based on the distribution of activated nodes and gap nodes in the semantic association network, the target language content associated with the activated nodes or the target language content corresponding to the gap nodes as the content to be presented in the immersive scene in the next step.

[0010] Optionally, the semantic association network construction module is further configured to: determine the physical objects in the immersive scene through image recognition, or read the electronic tags on the physical objects, to determine the presence of the referent; record a co-occurrence of the native language tag and the target language tag around the referent when the native language tag and the target language tag appear sequentially during the presence of the same referent and the time interval between their appearances does not exceed a set time window; when the number of co-occurrences of the native language tag and the target language tag reaches a set threshold, establish the two as a pair of mutually related nodes in the individualized cross-language semantic association network; when the association relationship determined by the number of co-occurrences is inconsistent with the translation relationship given by the preset bilingual dictionary, update the semantic association network with the association relationship determined by the number of co-occurrences.

[0011] Optionally, the code-switching detection module is further configured to: use a language recognition model to label the language to which each word in the child's mixed language speech belongs; determine the position where the language to which two adjacent words belong changes from the mother tongue to the target language, or from the target language to the mother tongue, as the code-switching point; for the word spoken in the target language at the code-switching point, extract contextual words in the mother tongue speech before and after the code-switching point that both refer to the same referent as the word spoken in the target language, and use the contextual words to determine the semantic node corresponding to the word spoken in the target language in the semantic association network, wherein the semantic node is marked as an activated node by the bidirectional semantic signal module.

[0012] Optionally, the training process of the language recognition model includes: using the acoustic features of the sample word as the input of the language recognition model; determining the output of the language recognition model as the language to which the sample word belongs; and using the language label of the sample word as the training label to train the language recognition model until the difference between the output and the language label converges.

[0013] Optionally, the immersive presentation module is further configured to: for concepts labeled as gap nodes, select the target language content corresponding to the gap node in the target language as the content to be presented in the next step in the immersive scene; for concepts labeled as activated nodes, determine the nodes to be presented that are associated with the activated nodes in the semantic association network; determine the nodes to be presented that are associated with both the native language tag and the target language tag as shared association nodes, and determine the nodes to be presented that are associated only with the target language tag as dedicated association nodes; first present the target language content corresponding to the shared association nodes, and then present the target language content corresponding to the dedicated association nodes.

[0014] Optionally, the immersive presentation module is further configured to: record the response time of the child from the end of the presentation of the target language content to the child's correct response to the target language content; determine the rolling average of the child's multiple response times within a set time period as the child's response time baseline; retain the nodes corresponding to the target language content with response times higher than the response time baseline in the presentation rotation of the immersive scene, and shorten the reproduction time interval of the nodes; and remove the nodes corresponding to the target language content with response times lower than the response time baseline from the presentation rotation.

[0015] Optionally, it also includes a mother tongue regression monitoring module, which is used to: record the immersive scene that evokes the target language concept for the target language concept that has been marked as an activated node; in the immersive scene that evokes the target language concept, record the detected expression of the target language concept by the child in their mother language as a mother tongue regression event; and immediately insert the activated node where the mother tongue regression event occurs into the head of the presentation queue of the immersive presentation module for reproduction, without waiting for the predetermined reproduction time of the activated node.

[0016] Optionally, the native language regression monitoring module is further configured to: determine that the current immersive scene and the immersive scene that evokes the target language concept are the same scene by matching the image features of the immersive scene or by determining the presence of the same referent; count the cumulative number of native language regression events occurring at the same activated node; when the cumulative number reaches a set number, increase the reproduction frequency of the activated node in the presentation queue; the native language regression event is recorded on the premise that the target language concept has been previously marked as an activated node, and concepts that have not been previously marked as activated nodes are not recorded.

[0017] Optionally, a fake friend disambiguation module is also included. This module is used to: determine the distance between the speech features of the target word and the speech features of the native word, and determine the reciprocal of the distance as the speech proximity between the target word and the native word; determine the number of common associated nodes between the target word and the native word in the semantic association network; when the speech proximity is higher than a set proximity threshold and the number of common associated nodes is lower than a set node number threshold, determine the target word and the native word as a fake friend risk pair; determine the scenario with the fewest associated nodes with the target word in the semantic association network, forcibly pair the target word with the scenario, and present it in the immersive scenario.

[0018] Optionally, the false friend disambiguation module is further configured to: traverse the target language words and native language words that the child has been exposed to, pair words whose phonetic similarity is higher than the set similarity threshold, and construct a cross-linguistic phonetic similarity word pair table; continuously monitor the number of common association nodes for word pairs in the cross-linguistic phonetic similarity word pair table; when the target language word and the native language word are determined to be a false friend risk word pair, and the child makes a correct response to the forced pairing presentation within a set number of times, the false friend risk word pair determination of the target language word and the native language word is lifted; when the number of common association nodes for word pairs in the cross-linguistic phonetic similarity word pair table reaches the set node number threshold, the word pair is removed from the false friend risk word pair.

[0019] The beneficial effects of this application are as follows: This application transforms the spontaneous code-switching that young children generate in immersive scenarios from interference that needs to be eliminated into directly usable semantic criteria. This allows the system to understand the cross-linguistic mastery of various concepts without requiring externally preset progress information. This is possible because the switching points between the child's native language and the target language are discrete and detectable events, naturally marking concepts that the child can already express in the target language and concepts that are still missing in the target language: a word spoken in the target language at a code-switching point indicates that the concept can be expressed in the target language, and this application marks the corresponding node in the semantic association network as an activated node; a concept expressed in the native language at a code-switching point indicates that the concept is still missing in the target language, and this application marks it as a missing node. Thus, the switching points discarded by existing processing become the basis for driving presentation. Based on the distribution of activated nodes and missing nodes, the system selects and presents related target language content or the target language content corresponding to the missing nodes, with the presentation direction directly corresponding to the child's current real state. For example, when a child utters a target language word in a sentence in their native language, the node corresponding to that word is marked as an activated node. Subsequently, the target language content associated with that node that has not yet been presented is prioritized to be completed, and the presentation is triggered by the child's own expression.

[0020] This application establishes cross-language correspondences without relying on a general bilingual dictionary. Instead, it establishes associations based on the co-occurrence frequency of native language and target language labels when the same referent is present, thereby correcting mismatches in general dictionaries for specific children. This effect works because bilingual labels appearing sequentially within a set time window around the same referent constitute natural anchor points for translational equivalence. That is, two labels that repeatedly co-occur when the same object is present refer to the same thing in the child's mind. This application accumulates such co-occurrences, and when the number of co-occurrences reaches a set threshold, it establishes a pair of associated nodes in the semantic association network. When the association relationship given by co-occurrence is inconsistent with the pre-set bilingual dictionary, co-occurrence takes precedence. Therefore, the specific referents of the same target word in different children's minds are recorded separately, and mismatches caused by general statistics are corrected at the individual level.

[0021] In addressing forgetting, this application uses spontaneous regression of a child's native language as an early criterion for forgetting, thus freeing the timing of retrieval from a fixed review order. For a target language concept marked as an activated node, if a child reverts to their native language expression in the same scenario that introduced the concept, this application records this as a native language regression event and immediately inserts the corresponding activated node at the head of the presentation queue for retrieval, without waiting for a predetermined retrieval time. This approach is based on the fact that a child's regression to their native language regarding a previously mastered concept often occurs before the review time determined by the system according to a fixed order, representing an observable early manifestation of forgetting. Therefore, retrieval is brought forward to the initial stage of forgetting, avoiding the problem of the fixed review order being later or earlier than the actual forgetting time.

[0022] This application identifies the risk of mispairing words with similar pronunciations but different meanings by using a cross-judgment of two conditions: phonetic similarity and semantic irrelevance. This overcomes the difficulty of distinguishing such mispairings by a single signal. Phonetic similarity alone cannot distinguish between normal translation and mispairing; semantic irrelevance alone cannot pinpoint the specific confusion caused by similar pronunciation. To address this, this application uses the reciprocal of the distance between the phonetic features of the target word and the native word as the phonetic similarity score. Furthermore, it counts the number of common association nodes in the semantic association network. Only when the phonetic similarity score is higher than a set similarity threshold and the number of common association nodes is lower than a set node count threshold is the word pair identified as a false friend risk. The target word is then forcibly paired with the scenario with the fewest association nodes. Thus, only word pairs that simultaneously satisfy phonetic similarity and weak semantic association are selected and presented in the scenario with the weakest association, helping young children re-establish the association between the target word and the correct meaning.

[0023] The above design elements work together to ensure that the system always progresses based on the information carried by the child's own real-time speech: code-switching provides direction for what has been mastered and what has not been mastered, physical co-occurrence provides individualized translation anchors, mother tongue regression provides early opportunities for forgetting, and the cross-judgment of speech and semantics provides a basis for identifying mis-binding; all of this information is taken from multimodal acquisition in immersive scenarios, without the need for additional manual annotation or external presets. Attached Figure Description

[0024] Figure 1 A schematic diagram illustrating an immersive bilingual interactive application scenario provided in an embodiment of this application; Figure 2 A block diagram of an immersive bilingual semantic association system for preschoolers provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a cross-language semantic association network provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the establishment of interconnected nodes for the physical co-occurrence provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the mechanism of code-switching bidirectional semantic signal processing provided in an embodiment of this application. Figure 6 A schematic diagram of the data flow for training the language recognition model provided in the embodiments of this application; Figure 7 This is a schematic diagram of the immersive content selection process provided in the embodiments of this application; Figure 8 A schematic diagram illustrating the response time baseline and reproduction adjustment provided for embodiments of this application; Figure 9 A schematic diagram illustrating the process of inserting the head of the queue for native language regression monitoring and presentation, as provided in this application embodiment; Figure 10 This is a schematic diagram illustrating the determination of dual signals for the false friend risk word pair provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This application provides an immersive bilingual semantic association system for preschoolers. The system operates at the level of natural language processing and semantic analysis, processing spontaneously generated mixed-language discourse in immersive bilingual scenarios. This includes discourse where individual target language words are inserted into a native language sentence, or where the target language is temporarily switched back to the native language when expressing a concept. Starting from the cross-linguistic semantic signals carried by such discourse, the system establishes and updates the individual-specific semantic correspondence between the native language and the target language in an individualized cross-linguistic semantic association network (hereinafter referred to as the semantic association network). It also schedules the target language content presented in the immersive scenario based on the node states of this network. The semantic association network is a data structure where nodes represent language labels and semantic concepts, and edges represent association relationships. It is gradually constructed from the preschooler's own real-time discourse and multimodal observations, rather than being derived from a general bilingual dictionary or pre-annotated static corpus.

[0027] At the implementation level, the system consists of four interconnected processing steps forming a closed loop: multimodal signal acquisition, semantic signal processing, association network maintenance, and content presentation. Multimodal signal acquisition captures children's speech signals, scene images, object information, and pointing or grasping actions. Semantic signal processing identifies the language and detects code-switching points in the speech signals, extracting the language switching locations in mixed speech as discrete signals. Association network maintenance updates the connections and states of nodes in the semantic association network based on the co-occurrence of bilingual tags when the pointed-to object is present and the semantic orientation on both sides of the code-switching points. Content presentation selects target language content based on the network state and outputs it in an immersive scene; the output results are then collected and returned to the processing stage. Thus, language switching in children's speech, which was previously treated as noise by existing processing, is transformed into computable signals that drive network updates and content presentation. This process involves the acquisition, recognition, and semantic association calculation of multimodal language data; its technical aspects lie in signal acquisition, language attribution determination, and semantic orientation localization, rather than relying on a pre-set schedule for the presented content.

[0028] Figure 1This is a schematic diagram illustrating an immersive bilingual interactive application scenario provided in an embodiment of this application. For example... Figure 1 As shown, the immersive scene includes a data acquisition module and an immersive presentation module. The data acquisition module captures the child's speech, images, objects, and pointing or grasping actions. Objects in the scene may be tagged with electronic tags. In this application, the pointed-to object refers to an object in the current immersive scene that is pointed to or grasped by the child, or identified as present by image recognition and electronic tags. The acquired multimodal signals are sent to an individualized cross-language semantic association network for processing. The processing results are converted into target language content by the immersive presentation module and re-presented in the same immersive scene. Thus, data acquisition, processing, and presentation are interconnected. In some embodiments, the data acquisition module uses a sampling rate of 16 kHz to 48 kHz for speech acquisition, typically 16 kHz, to continuously record the child's speech. Image acquisition captures scene images at a frame rate of several to tens of frames per second. Pointing or grasping actions are acquired through visual or tactile sensors in the scene. All three signals have a unified timestamp for subsequent time alignment.

[0029] Figure 2 A block diagram of an immersive bilingual semantic association system for preschoolers provided in an embodiment of this application. Figure 2 As shown, the system includes a data acquisition module, a semantic association network construction module, and a code-switching detection module. It also includes a bidirectional semantic signal module and an immersive presentation module. These modules work collaboratively on an individualized cross-lingual semantic association network. In some embodiments, the system further includes a mother tongue regression monitoring module and a fake friend disambiguation module. The output of the acquisition module is sent to the semantic association network construction module and the code-switching detection module. The semantic association network construction module maintains nodes and associations in the network based on the co-occurrence of bilingual tags when the referent is present. The code-switching detection module locates code-switching points in mixed discourse. The bidirectional semantic signal module receives the results from the code-switching detection module and marks activated and missing nodes in the network. The immersive presentation module selects and outputs target language content based on the distribution and status of nodes in the network. All modules read and write around the same semantic association network, forming the aforementioned closed loop.

[0030] The method provided in this application runs on the aforementioned system, and its overall process includes the following steps: collecting multimodal signals from young children in an immersive scenario; establishing and updating interconnected nodes in a semantic association network based on the co-occurrence of native language tags and target language tags when the referent is present; determining code-switching points from mixed discourse and marking the semantic pointers on both sides of the code-switching points in the network as activated nodes or gap nodes; selecting target language content based on the distribution of activated nodes and gap nodes and presenting it in the immersive scenario; collecting the children's responses to the presented target language content and adjusting subsequent presentations accordingly. The aforementioned steps are executed cyclically during system operation, and the nodes and associations in the network continuously evolve with the accumulation of children's discourse. Native language tags refer to language tags recorded in native language characters or phonetics that point to a specific referent or concept, while target language tags refer to corresponding tags recorded in the target language.

[0031] Before entering subsequent processing, the pre-processed speech signals of young children are converted into word sequences with time boundaries through speech preprocessing. In one embodiment, speech preprocessing employs lightweight speech recognition or keyword detection to segment individual words from continuous speech, providing the time boundary and morphology of each word. The morphology, after normalization, serves as the native language label or target language label corresponding to the word. Thus, language identification can be performed by labeling the language at the word level, and the construction of the semantic association network can establish associations using the labels corresponding to the morphology as nodes. The calculation of speech proximity and the localization of semantic nodes also use the word's identity as input. The reason for segmenting words and normalizing their morphology first is that co-occurrence network construction, semantic localization, and speech distance calculation all rely on the word's identity; without language identification and morphology, it is impossible to determine which pair of labels participates in co-occurrence.

[0032] The semantic association network construction module is responsible for establishing associations between native language tags and target language tags based on the co-occurrence of objects in a child's own context, without relying on a general bilingual dictionary. The fundamental basis is that native language tags and target language tags that appear sequentially around the same referent at similar times constitute a natural anchor point for translational equivalence. That is, two tags that repeatedly co-occur when the same object is present refer to the same thing in the child's context. Therefore, this application uses the cumulative number of such co-occurrences as the criterion for establishing associations, rather than using general translations obtained from group statistics. The reason for this is that the translations provided by general dictionaries are static mappings oriented towards the group; the specific referent of the same target word often differs among different children. If a general translation is directly used, it cannot reflect the child's actual referent, nor can it correct for any discrepancies.

[0033] There are two ways to determine the presence of the pointed object. In one embodiment, the acquisition module identifies the physical object in the immersive scene through image recognition, that is, it performs target detection and classification on the scene image, outputs the category and position of the physical object in the screen, and determines the currently present pointed object. In other embodiments, the physical object is attached with an electronic tag, and the acquisition module determines the presence of the pointed object by reading the object identifier in the electronic tag in the near field. This method does not depend on lighting or occlusion conditions and is suitable for situations where the physical object is easily held and obscured by young children. The two methods can be used individually or in combination, and the category output of the image recognition is corrected by the read electronic tag identifier. The period of presence of the same pointed object refers to the continuous time from the moment the pointed object is determined to be present until it leaves the screen, or the electronic tag reading is interrupted and continues for more than the set departure determination time. The departure determination time can be 0.5 seconds to 3 seconds, typically 1 second.

[0034] The aforementioned image recognition is implemented using an image recognition model. This model takes a scene image as input and outputs the category labels and locations of objects within the image, for example, a lightweight convolutional network designed for mobile devices. Its samples are images of common objects in immersive scenes. The features of the samples are the image pixels and their visual features extracted through convolution. The labels are the object categories and bounding boxes, manually or semi-automatically labeled. In one embodiment, the image recognition model uses a lightweight convolutional neural network as its backbone, with millions to tens of millions of parameters, enabling real-time inference on edge computing power at the scene side. The image recognition model is trained using cross-entropy loss to constrain the category output and regression loss to constrain the location output until the recognition error on validation samples stabilizes. To address cold starts, the image recognition model is initialized with a detection model pre-trained on general object images, and then fine-tuned with a small number of scene-specific object images. During system operation, newly appearing objects are automatically labeled with electronic tags, thus incrementally updating the image recognition model without requiring the re-collection of large-scale samples.

[0035] Figure 4 This is a schematic diagram illustrating the establishment of interconnected nodes for the physical co-occurrence provided in the embodiments of this application. For example... Figure 4 As shown, during a period of time when the referent is present, the native language label and the target language label appear sequentially along the time axis. This application defines a co-occurrence of the native language label and the target language label around the referent as a single occurrence of the native language label and the target language label around the referent when the native language label and the target language label appear sequentially during the same period when the referent is present, and the time interval between their appearances does not exceed a set time window.

[0036] In one embodiment, the native language tags and target language tags participating in co-occurrence counting are limited to words that appear closest to the time of the object being referred to, or that accompany the child's pointing or grasping action towards the object. These are words identified as naming the object through spatiotemporal alignment of image and speech. Words that only co-occur with the object in time, but do not point to the object, are not included in the co-occurrence count surrounding the object. This limitation is based on the consideration that if only temporal co-occurrence is considered, frequently occurring words in the same sentence that are unrelated to the object being referred to would be consistently miscounted as co-occurring, making them impossible to filter out by increasing the set threshold.

[0037] A time window is set to limit the maximum time span during which two tags are considered to co-occur around the same referent. This time window can range from 1 to 10 seconds, typically 4 seconds. If the time window is too short, bilingual tags spoken sequentially while the same referent is present may be missed due to the slightly larger interval, resulting in slow association establishment. If the time window is too long, two tags that span different referents but are only coincidentally adjacent in time may be mistakenly counted as co-occurring, introducing incorrect associations. In some embodiments, the time window is adaptively adjusted based on the child's average speaking speed and the average duration of the referent's presence; when the speaking speed is slow or the referent's presence is long, the larger end of the interval is used.

[0038] like Figure 4 As shown, this application accumulates the number of co-occurrences of the same pair of native language tags and target language tags around the same referent. When the number of co-occurrences reaches a set threshold, the two are established as a pair of related nodes in the individualized cross-linguistic semantic association network. The set threshold is used to suppress occasional, unstable co-occurrences from being mistakenly established as associations. Its value can be 3 to 8 times, typically 5 times. If the set threshold is too small, one or two accidental co-occurrences will be solidified as associations, making it susceptible to noise interference; if the set threshold is too large, the association establishment will be slow, and the individualized correspondence will not be formed for a long time. In some embodiments, the co-occurrence count adopts an accumulation method with forgetting, that is, the count of co-occurrences that have not occurred for a long time slowly decreases according to a set decay coefficient. This decay coefficient can be 0.9 to 0.99 per day, typically 0.95, so that occasional associations that have not occurred for a long time do not occupy the threshold for a long time.

[0039] In one embodiment, nodes in the semantic association network are divided into two levels: label nodes and sense nodes. Label nodes correspond to the word form of a native or target language, while sense nodes correspond to the specific semantic meaning of that word form in a particular referent or context. When the same target language word form repeatedly co-occurs with different native language labels in the presence of different referents, multiple sense nodes corresponding to each referent are differentiated under the target language label node; if it co-occurs only with a single referent, only one sense node is under the target language label node. Therefore, the node corresponding to the target language label in the network is the sense node under that label node, and the aforementioned activated nodes and gap nodes are all labeled at the sense node level. The reason for distinguishing between label nodes and sense nodes is that the accessibility of different senses of a polysemous target language word may differ for the child. Only by reaching the sense node can bidirectional semantic signals and subsequent presentation process different senses of the same word form separately.

[0040] In some embodiments, the semantic association network construction module also includes a pre-built bilingual dictionary to provide initial candidate correspondences when co-occurrence data is insufficient. For example... Figure 4 As shown, when the association determined by the number of co-occurrences is inconsistent with the translation relationship given by the pre-built bilingual dictionary, the semantic association network is updated with the association determined by the number of co-occurrences. That is, the dictionary's group translation is covered based on the actual co-occurrence of the child. The reason for this approach is that the pre-built bilingual dictionary provides a group-oriented statistical translation, while co-occurrence reflects the child's actual referent in a specific scenario. When the two conflict, the individual's actual referent is more likely to reflect the true semantic correspondence in that child's context. If the dictionary is still the standard, the aforementioned individualized correction will not be possible. Thus, the mismatch caused by general statistics is corrected at the individual level, and the specific referents of the same target word in different children's contexts are recorded separately.

[0041] The code-switching detection module is used to identify code-switching points between the native language and the target language in children's mixed language discourse. Figure 5 This is a schematic diagram illustrating the mechanism of code-switching bidirectional semantic signal processing provided in an embodiment of this application. For example... Figure 5 As shown, the mixed discourse is labeled with the language of each word by a language recognition model, resulting in a string of language labels arranged by word. The position where the language of two adjacent words changes from the native language to the target language, or vice versa, is determined as the code-switching point. The code-switching point is the boundary where a child switches from one language to another within a sentence; it is a discrete and detectable event that marks a boundary of the child's cross-language semantic state. The reason for using the language jump between adjacent words as the code-switching point is that this jump position corresponds precisely to the moment when the child switches from one language to another in expression, with the concepts on both sides carrying the concepts that can be expressed in the target language and those that still rely on the native language, respectively.

[0042] The bidirectional semantic signal module labels two types of nodes in the semantic association network based on the semantic orientation on both sides of the code-switching point. For words spoken in the target language at the code-switching point, the corresponding nodes in the semantic association network are labeled as activated nodes. For concepts that the child expresses in their native language at the code-switching point, the corresponding nodes in the target language are labeled as gap nodes. Activated nodes represent nodes corresponding to concepts that the child can already express in the target language, while gap nodes represent nodes corresponding to concepts that the child is not yet able to express in the target language and can only express in their native language. Figure 3 This is a schematic diagram of the structure of a cross-language semantic association network provided in an embodiment of this application. Figure 3 As shown, in the cross-linguistic semantic association network, activated nodes are represented by double rings, and missing nodes are represented by dashed rings. Both are nodes corresponding to the target language label in the semantic association network, the difference being their state of being actually expressed by the child in the target language. Thus, the same network simultaneously carries the semantic states of both the child's ability to express themselves in the target language and the child's lack of expression, and the code-switching points thus provide bidirectional semantic signals to the network.

[0043] The determination of gap nodes is symmetrical to that of activated nodes. For concepts that a child expresses in their native language at code-switching points, the native language tags describing the same referent in the local and neighboring native language discourse are used to locate the corresponding native language-side node in the semantic association network. If a corresponding target language tag node already exists in a pre-set bilingual dictionary or existing corpus, but the child has not yet produced the concept in the target language, then the existing target language tag node is marked as a gap node. If no corresponding target language tag node exists, a new placeholder target language tag node is created and marked as a gap node, with its word form to be produced by the child in the target language later, or supplemented by the presented content. Therefore, gap nodes mark concepts that the child has not yet produced in the target language, but which already have a corresponding native language side in their semantic association network. This does not conflict with the subsequent targeted presentation of the target language content for gap nodes. The target language content used for targeted presentation is taken from dictionaries or existing corpora elsewhere, while the gap refers to the state where the child has not yet produced the concept in the target language.

[0044] In existing cross-language processing, to obtain clean monolingual input, language switching in mixed discourse is often treated as noise removal or normalization, resulting in the loss of code-switching points. This application does not remove these switches but retains them and extracts them as signals. The basis for this approach is that code-switching points, when expressed in the target language, point to concepts that the child can already express in the target language; code-switching points, when changed back to concepts expressed in the native language, point to concepts that the child is still missing in the target language. If the approach of removing switches is adopted, the system will neither record the fact that the child can express a certain concept in the target language nor know that a certain concept is still missing in the target language. The presented content can only proceed based on an externally preset vocabulary list, rather than based on the child's current actual cross-language state. Therefore, the detection and annotation of code-switching points is essentially extracting and structuring the child's cross-language semantic state from multimodal language data.

[0045] In some embodiments, when the language identification model labels the language of each word, it simultaneously outputs the confidence score of the label. This confidence score threshold can be between 0.6 and 0.9, typically 0.8. When the confidence score of a word's language label is lower than this threshold, the model combines the word's existing attribution in the semantic association network to assist in the determination, avoiding false detections caused by homographs or target words already borrowed from the native language. Optionally, a code-switching point is only identified when the language transition between adjacent words lasts for at least one complete word and the language labels before and after the transition are stable, thereby suppressing false transitions caused by misidentification of single syllables. This approach is based on the consideration that mixed discourse occasionally contains ambiguous pronunciations or borrowed words, and single-point language labeling may be incorrect. If every adjacent labeling difference is indiscriminately identified as a code-switching point, a large number of false activated or missing labels will be introduced into the network.

[0046] For a word spoken in the target language at a code-switching point, this word may correspond to multiple semantic nodes in the semantic association network, meaning the same target word has multiple semantic references. To determine the specific semantic node being referred to, the code-switching detection module extracts contextual words from the native language discourse before and after the code-switching point that, along with the word spoken in the target language, point to the same referent. These contextual words are then used to determine the corresponding semantic node for the target word in the semantic association network, and this semantic node is marked as an activated node. Contextual words are native language words describing the same referent in the adjacent native language discourse at the code-switching point; they are determined by the presence of the referent and the temporal proximity of the discourse. For example... Figure 5 As shown, after determining the semantic node from the contextual word, the corresponding semantic node is marked as an activated node.

[0047] The determination of semantic nodes employs the intersection criterion of associated neighborhoods. Specifically, each candidate semantic node of the target word in the semantic association network is associated with a set of nodes, and the context word also corresponds to a set of associated nodes in the network. The intersection of the associated node set of each candidate semantic node and the associated node set of the context word is calculated, and the candidate semantic node with the largest intersection is taken as the semantic node referred to in this context. This approach is based on the fact that the semantic node with the most overlap with the current context word's associated neighborhood is precisely the semantic node that best fits the meaning of the current referent. In one embodiment, when the intersection sizes of two candidate semantic nodes are similar and difficult to distinguish, the category of the physical object present in the referent is further used as a constraint, and candidates whose semantic node's associated neighborhood contains the current referent are selected. If they still cannot be distinguished, the activated node is temporarily not marked, and subsequent co-occurrence supplementary evidence is reserved to avoid mislabeling. Thus, the disambiguation of the semantics of polysemous target words is based on the intersection operation of associated neighborhoods in the network, without the need for external explanatory resources.

[0048] Language identification models are used to label the language of each word in a child's mixed speech, which is the basis for code-switching detection. Figure 6 This is a schematic diagram of the data flow for training the language recognition model provided in an embodiment of this application. For example... Figure 6 As shown, the language identification model takes the acoustic features of a sample word as input and the language to which the word belongs as output. In one embodiment, the acoustic features are composed of Mel-frequency cepstral coefficients extracted from the speech signal after framing. The frame length is 20 to 40 milliseconds, typically 25 milliseconds; the frame shift is 8 to 15 milliseconds, typically 10 milliseconds; and each frame takes 13 to 40 dimensions of cepstral coefficients. The features of each frame covered by the word are aggregated according to the word's temporal boundary to form the word's acoustic feature sequence. The extraction of acoustic features is a deterministic signal processing method, without trainable parameters. Its function is to transform the original speech signal into a representation that is sensitive to language differentiation and relatively robust to speaker differences.

[0049] In one embodiment, the language identification model is implemented using a temporal modeling network. Its structure includes several layers of one-dimensional convolutional or bidirectional gated recurrent units, such as a time-delay neural network or a lightweight recurrent convolutional network. This temporal modeling network models the temporal context of the acoustic feature sequence and terminates with a binary classification output layer to determine whether the word belongs to the native language or the target language. The language identification model has a parameter count between 100,000 and 1 million, making it a lightweight model that can perform word-by-word inference at a speed exceeding real-time on edge computing power at the scene side. The reason for using a lightweight temporal network instead of a large model is that the number of categories for word-by-word language identification is small, and the context span is short; a lightweight model is sufficient to achieve the required accuracy while facilitating rapid updates on individual data.

[0050] like Figure 6As shown, the language identification model is trained using the language labels of sample words as training labels. Training samples come from two sources: firstly, publicly available bilingual speech corpora, whose language labels are given with the corpus; and secondly, from children's speech accumulated during the system's operation, whose language labels are obtained from the co-occurrence relationships of bilingual labels when the referred object is present, as well as a small number of user-confirmed annotations, thus expanding the individual samples without requiring manual annotation word by word. During training, the acoustic features of the sample words are input into the language identification model to obtain the language discrimination output. The difference between this output and its corresponding language label is calculated, and this difference is measured using binary cross-entropy loss. Figure 6 The difference between the output and its corresponding language label is used to adjust the model parameters in reverse. The model is considered complete when this difference no longer decreases on the validation samples. Figure 6 Training ends when the convergence criterion is yes; otherwise, iteration continues. In one embodiment, the convergence criterion is that the relative change in classification error on the validation set is less than a set small amount within consecutive set rounds.

[0051] During the inference phase, the language identification model takes the acoustic features of words as input and the language to which the word belongs as output, such as... Figure 6 As the explanatory text indicates, the system provides word-by-word language discrimination for the child's real-time speech, enabling the code-switching detection module to locate code-switching points. To address cold start, i.e., when individual samples are insufficient, the language recognition model is initialized with a model pre-trained on a general bilingual speech corpus, initially possessing basic native and target language discrimination capabilities. It is then fine-tuned incrementally as the child's speech accumulates, gradually adapting to the child's pronunciation characteristics. During system operation, the language recognition model is re-fine-tuned periodically with newly accumulated individual samples, or triggered for updates when an overall decrease in discrimination confidence is detected, thereby tracking changes in the child's pronunciation as they grow. Thus, the language recognition model, from acoustic feature input to language discrimination output, from training labels to convergence criteria, and from inference to incremental updates, is all implemented in a way that can be reproduced by those skilled in the art.

[0052] The immersive presentation module selects the target language content to be presented in the immersive scene next, based on the distribution of activated and missing nodes in the semantic association network. This selection has two directions: for missing nodes, it selects the corresponding target language content to fill in the gaps in the child's understanding of the target language; for activated nodes, it selects target language content associated with those nodes to expand concepts already expressed in the target language into their semantic neighborhood. Target language content refers to target language corpora that can be presented in the immersive scene and correspond to a specific node in the network, such as physical objects, images, or audio clips related to the target language. Therefore, the presentation direction directly corresponds to the child's current semantic state, without relying on an externally preset fixed order.

[0053] Figure 7This is a schematic diagram illustrating the immersive content selection process provided in an embodiment of this application. For example... Figure 7 As shown, in step S1, the immersive rendering module reads the annotations of nodes in the semantic association network and determines whether the currently processed node is labeled as a gap node or an activated node. The concept of a node labeled as a gap node is as follows: Figure 7 As shown in the branch of the missing node, in step S2, the target language content corresponding to the missing node in the target language is selected as the content to be presented in the immersive scene in the next step. For the concept of a node marked as activated, such as... Figure 7 As shown in the activated node branch, in step S3, the node to be presented that is associated with the activated node is determined in the semantic association network, that is, the target language node that has an associated edge with the activated node and has not yet been presented recently.

[0054] like Figure 7 As shown, in step S4, nodes that are associated with both the native language tag and the target language tag are identified as shared-association nodes; nodes that are associated only with the target language tag are identified as exclusive-association nodes. Shared-association nodes are nodes in the child's semantic association network that are simultaneously attached to both the native language tag and the target language tag, and their semantics are already supported by the native language side; exclusive-association nodes are nodes that are only attached to the target language tag and do not yet have a corresponding node on the native language side. For example... Figure 3 As shown, the shared association node is located at the intersection of the association between the native language tag and the target language tag, while the exclusive association node is only connected to the target language tag.

[0055] like Figure 7 As shown, in step S5, the target language content corresponding to the shared association node is presented first, followed by the target language content corresponding to the proprietary association node. The reason for presenting the shared association node first and then the proprietary one is that the shared association node already has semantic support on the native language side. When transitioning from an activated node to a shared association node, children can use existing native language connections to understand new target language content, forming a smooth transition from a mastered concept to its semantic neighborhood. The proprietary association node, however, does not yet have a corresponding counterpart on the native language side. If presented first, it lacks existing connections to rely on, resulting in higher comprehension costs. Therefore, the presentation order progresses from dense to sparse along the intersection of association neighborhoods in the semantic association network, starting with the activated node and establishing connections step-by-step from the shared association node to the proprietary association node, ensuring that the expansion of the target language content always has existing semantic support.

[0056] After presenting the target language content, the immersive presentation module further adjusts the subsequent presentation based on the children's response to the content. Figure 8 This is a schematic diagram illustrating the response time baseline and reproduction adjustment provided in an embodiment of this application. Figure 8As shown, the horizontal axis represents time, and the vertical axis represents response duration. The immersive presentation module records the response duration from the end of the presentation of the target language content to the time when the child makes a correct response to that content. Response duration is the time interval between the end of the presentation and the time when the correct response is elicited by the target language content. A correct response refers to the child's identification, grasping, or repetition in the scene that is semantically consistent with the target language content. Whether the identification or grasping is correct is determined by whether the identified or grasped object corresponds to the object referred to in the target language content; whether the repetition is correct is determined by comparing the child's speech with the target language word form after the aforementioned word segmentation and recognition to see if a set similarity is achieved; the start and end times of the response are determined by the acquisition module, starting from the end of the presentation and ending at the time when a correct response is detected. Response duration is a time quantity that can be directly measured by the acquisition module, reflecting the child's current accessibility to the target language content.

[0057] like Figure 8 As shown in the figure, the solid line represents the successively measured response duration, and the horizontal dashed line represents the response duration baseline. The rolling average of the children's response durations over a set period is determined as the children's response duration baseline. The response duration baseline is a reference level obtained by taking the rolling average of response durations over a set period. It is continuously updated with new responses, representing the overall level of the child's recent response speed. The set period is used to limit the time range of responses participating in the rolling average, and its value can be 1 to 7 days, typically 3 days; correspondingly, the number of responses participating in the rolling average is the most recent 8 to 30, typically 16. If the set period is too short, the baseline will be significantly affected by individual abnormal responses, resulting in large fluctuations; if the set period is too long, the baseline will be slow to track the child's recent changes. The reason for using the baseline obtained by rolling average rather than a fixed time limit is that the absolute values ​​of response durations vary greatly among different children and for different content. Using the individual's own recent level as a reference is more effective in reflecting relative changes than a uniform absolute time limit.

[0058] like Figure 8 As shown, for target language content with a response time longer than the baseline response time, the corresponding node is retained in the presentation cycle of the immersive scene, and the reappearance interval of that node is shortened. The reappearance interval is the time interval between two presentations of the target language content corresponding to the same node; shortening the reappearance interval, for example, to 30% to 60% of the original reappearance interval, typically 50%. If the shortening is too small, the reappearance encryption is insufficient, and content with a slow response will not be reproduced in time; if the shortening is too large, the content will be reproduced too frequently, crowding out the presentation opportunities of other content. The basis for this treatment is that a response time longer than the baseline indicates that the child's response to the target language content is relatively slower, which is an observable manifestation of a decrease in its semantic accessibility. Therefore, its reappearance is encrypted so that the content is presented again in the early stages of the decrease in accessibility.

[0059] like Figure 8 As shown, for target language content with a response time no longer than the baseline response time, the corresponding node is removed from the presentation rotation. The reason for removing content with a response no slower than the baseline is that its response is at or faster than the child's recent level, its semantic accessibility is relatively stable, and the benefit of continuing to occupy the presentation rotation is limited. Removing it frees up limited presentation opportunities for content with a slower response that requires more reproducibility. Therefore, the presentation rotation is dynamically adjusted based on the relative speed of the child's real-time response; content with a slower response is reproduced with enhanced encryption, while content with a stable response is removed, and presentation resources are automatically allocated according to the child's semantic accessibility.

[0060] In some embodiments, the system further includes a mother tongue regression monitoring module for adjusting the reproduction of activated nodes based on the child's mother tongue regression. Figure 9 This is a schematic diagram illustrating the process of inserting the head of the queue for native language regression monitoring and presentation, as provided in an embodiment of this application. Figure 9 As shown, in step S1, for the target language concept marked as an activated node, the immersive scene that evoked the target language concept is recorded, that is, the scene in which the child previously expressed the concept in the target language. In step S2, by matching the image features of the immersive scene or by determining the presence of the same referent, it is determined that the current immersive scene and the immersive scene that evoked the target language concept are the same scene. The reason for determining that they are the same scene is that children's expressions in different scenes may be different. Only by re-observing in the same scene where the concept was previously evoked and using the expression in their native language can the change in the accessibility of the concept be used as a criterion.

[0061] like Figure 9 As shown, in step S3, under the condition that it is determined to be the same scene, the detected expression of the target language concept by the child in their native language is recorded as a native language regression event. A native language regression event is an event in which a child, in the same scene where they previously expressed a concept in the target language, reverts to expressing the concept in their native language. The recording of this event is based on the premise that the target language concept has been previously marked as an activated node. For concepts that have not been previously marked as activated nodes, even if they are detected as being expressed in the native language, they are not recorded as native language regression events. The consideration for this approach is that native language regression aims to capture situations where the child was able to express the concept in the target language but subsequently reverted to their native language; without this premise, concepts that the child encounters for the first time and can only be expressed in their native language would be mistakenly recorded as regressions, and the regression event would lose its indicative significance.

[0062] like Figure 9As shown, in step S4, the activated node that has experienced a mother tongue regression event is immediately inserted at the head of the presentation queue of the immersive presentation module for playback, without waiting for the predetermined playback time of the activated node. The presentation queue is the queue of nodes to be played in a predetermined order in the immersive presentation module, with the head of the queue being the first position to be played. The reason for immediately inserting it at the head of the queue without waiting for the predetermined playback time is that when a child regresses to their mother tongue based on a previously mastered concept, it often occurs before the playback time determined by a fixed order. This is an early observable manifestation of the decline in the accessibility of the concept. Therefore, playing it earlier ensures that the activated node is re-presented at the beginning stage of the decline in the accessibility of its corresponding concept, avoiding the fixed playback order being later or earlier than the actual change time.

[0063] like Figure 9 As shown, in step S5, the cumulative number of times the same activated node experiences a mother tongue rollback event is counted. When the cumulative number reaches a set number, as shown in step S6, the reappearance frequency of the activated node in the presentation queue is increased. This set number is used to distinguish between an occasional rollback and recurring rollbacks; its value can range from 2 to 5 times, typically 3 times. If the set number is too small, a single occasional rollback will trigger a continuous increase in the reappearance frequency, easily leading to over-reappearance; if the set number is too large, repeated rollbacks will not be reflected in the reappearance frequency for a long time, resulting in a lag in adjustment. Therefore, mother tongue rollback monitoring triggers immediate queue head reappearance with a single rollback and a continuous increase in the reappearance frequency with a cumulative number of rollbacks, responding separately to both occasional and recurring scenarios.

[0064] In one embodiment, step S2 determines the matching of image features for the same scene by using the similarity between global visual features extracted from the scene images. When the similarity is higher than a set scene similarity threshold, the scene is determined to be the same. This scene similarity threshold can be 0.7 to 0.9 after the similarity is normalized to 0 to 1, typically 0.8. To determine if the same object is present, the identification of the object obtained through image recognition or electronic tag reading is checked for consistency. These two methods can be used individually or in combination. When there are significant changes in lighting and viewing angle, the identification of the object is prioritized to avoid misjudging the same scene based solely on image feature similarity. This approach allows the determination of the same scene to be based on both the overall image and the actual objects present, adapting to the changing conditions of both the image and the objects in an immersive scene.

[0065] In some embodiments, the system also includes a fake friend disambiguation module for identifying and handling misclassification of cross-linguistic words that are phonetically similar but semantically unrelated. Figure 10 This is a schematic diagram illustrating the dual-signal determination of false friend risk words provided in an embodiment of this application. Figure 10As shown, the horizontal axis represents speech proximity, and the vertical axis represents the number of common associated nodes. The fake friend disambiguation module determines the distance between the speech features of the target word and the speech features of the native word, and the reciprocal of this distance is determined as the speech proximity between the target word and the native word. In one embodiment, the reciprocal of this distance is normalized to 0 to 1 according to the high quantile of the proximity in the vocabulary. When the distance approaches 0, it is truncated with the upper bound corresponding to the high quantile to obtain the normalized speech proximity. Speech proximity is a measure of the similarity between the pronunciation of the target word and the native word; the smaller the distance, the higher the proximity. In one embodiment, the speech features are taken from the aforementioned Mel-frequency cepstral coefficient sequence, and the distance between them is measured by dynamic time-warped distance to allow for different word durations; the calculation of speech proximity is a deterministic operation and does not contain trainable parameters.

[0066] The false friend disambiguation module also determines the number of common association nodes between the target word and the native word in the semantic association network. The number of common association nodes is the number of elements in the intersection of the set of association nodes corresponding to the target word and the set of association nodes corresponding to the native word, reflecting the degree of semantic association between the two words. For example... Figure 10 As shown, when the speech proximity is higher than a set proximity threshold and the number of shared nodes is lower than a set node count threshold, the target word and the native word are identified as a false friend risk word pair. Figure 10 The area to the right of the vertical dashed line and below the horizontal dashed line is represented by a solid dot; word pairs whose voice proximity is not higher than the set proximity threshold, or whose number of common related nodes is not lower than the set number of nodes threshold, are represented by hollow dots, which are non-fake friend risk word pairs.

[0067] False friend risk word pairs are cross-linguistic word pairs that are phonetically similar but semantically weak. A proximity threshold is set to define the degree of sufficient phonetic similarity. After normalizing the phonetic similarity to 0-1, a value of 0.6 to 0.85 is suitable, with 0.75 being typical. If the proximity threshold is set too low, many words with only slight phonetic similarities will be included, leading to numerous misclassifications; if it is set too high, truly easily confused phonetically similar words will be missed. A node number threshold is set to define the degree of weak semantic connection. A value of 1 to 3 nodes is suitable, with 2 being typical. If the node number threshold is set too high, words with some normal connection will be misclassified as risk word pairs; if it is set too low, mispairings with extremely weak semantic connection will be difficult to include. The reason for using the intersection of two conditions, phonetic similarity and weak semantic association, is that phonetic similarity alone cannot distinguish between normal translation and mismatch, and weak semantic association alone cannot locate the confusion caused by similar pronunciation. Only word pairs that satisfy both conditions are at risk of mismatch due to similar pronunciation and different meanings.

[0068] For target words identified as risky "fake friend" pairs, the fake friend disambiguation module determines the scene with the fewest associated nodes with the target word in the semantic association network, forcibly pairs the target word with that scene, and presents it in the immersive scene. In one embodiment, each immersive scene is represented as a scene node in the semantic association network, and the scene node is associated with the referent and its tag nodes that appear in the scene; the number of associated nodes between a scene and the target word is the size of the intersection of the set of associated nodes of the node corresponding to the target word and the set of associated nodes of the scene node. Based on this, all registered scenes are traversed, and the scene with the smallest intersection, i.e., the fewest associated nodes with the target word, is selected as the scene for forced pairing. In one embodiment, the aforementioned traversal is performed within the scope of scenes containing the correct referent of the target word or containing the native language tag corresponding to the correct referent, thereby ensuring that there is a correct referent in the scene that can be used to reconstruct the association even when the existing association is weakest.

[0069] The reason for selecting the scenario with the fewest associated nodes is that the existing semantic association between the scenario and the target word is the weakest. Presenting the target word in such a scenario minimizes interference from existing incorrect associations, making it easier to re-establish the association between the target word and the correct referent in the scenario. If the target word is presented in a scenario where it already has a strong association with the word, the existing incorrect associations will be further strengthened.

[0070] In some embodiments, the pseudo-friend disambiguation module also iterates through the target language words and native language words that the child has already encountered, pairing words with a phonetic similarity higher than a set similarity threshold to construct a cross-lingual phonetic similarity word pair table. This table is constructed incrementally offline or when the system is idle; that is, phonetic similarity is calculated and paired only for newly encountered words with existing words, rather than recalculating the entire list each time. In one embodiment, words are first mapped to several coarse bins based on acoustic features, and phonetic distance is calculated only between words in the same or adjacent bins, thus preventing the computational load from increasing twice with the number of words. The cross-lingual phonetic similarity word pair table consists of cross-lingual word pairs with similar pronunciations, used to centrally register phonetic similarity word pairs that require continuous monitoring. For word pairs in this table, the pseudo-friend disambiguation module continuously monitors the number of common associated nodes to track changes in the semantic association of each phonetic similarity word pair. The rationale behind this approach is that the phonetic similarity remains relatively stable once the two words are identified, while the number of common association nodes changes continuously as children accumulate their speech. By using phonetic similarity as a one-time screening criterion and the number of common association nodes as a continuous monitoring metric, changes in phonetic similarity word pairs from weak association to normal association, or vice versa, can be detected in a timely manner without repeatedly calculating phonetic distance.

[0071] For the word pairs in the table, the false friend disambiguation module further adjusts its judgment based on two types of conditions. When the target word and the native word are judged as a false friend risk pair, and the child makes consecutive correct responses to the aforementioned forced pairing presentation within a set number of times, the false friend risk pair judgment for the target word and the native word is lifted. The set number of times here is used to confirm that the child can stably and correctly distinguish the word pair in the forced pairing scenario, and its value can be 3 to 6 times, typically 4 times; if the set number is too small, the judgment may be lifted too early if there is only occasional consecutive correct responses, while if it is too large, the lifting may be delayed, and the word pair will remain in the forced pairing presentation for a long time. Figure 10 As shown, when the number of common associated nodes of a word pair in the cross-language homophone pair table reaches a set threshold, the word pair is removed from the list of risky fake friend word pairs. Figure 10 The middle word pair moves upward from below the horizontal dotted line and crosses it. Therefore, the determination of a false friend risk word pair is not a one-time event, but is appropriately removed as the child accumulates correct responses and the semantic association strengthens.

[0072] Based on the same inventive concept, this application also provides system embodiments corresponding to the foregoing methods. (Refer to...) Figure 2 The system comprises multiple modules corresponding one-to-one with the processing parts of the aforementioned methods. Specifically, the acquisition module acquires multimodal signals; the semantic association network construction module determines the presence of the referred object, counts co-occurrences, and establishes interconnected nodes; the code-switching detection module identifies word-by-word categories, locates code-switching points, and determines semantic nodes; the bidirectional semantic signal module labels activated and missing nodes; and the immersive presentation module selects, rotates, and adjusts the presentation of target language content. Each module revolves around individualized cross-linguistic semantic association network reading and writing, with the signal flow between them as follows: Figure 2 As shown.

[0073] In some embodiments, refer to Figure 2 The system also includes a native language regression monitoring module and a false friend disambiguation module. The native language regression monitoring module records the elicitation scene, determines the same scene, and records and counts native language regression events, as well as inserting the head of the queue of activated nodes and adjusting the reproduction frequency. The false friend disambiguation module calculates the speech proximity and the number of commonly associated nodes, determines the false friend risk word pairs, selects the forced pairing scene, and constructs and maintains the cross-language near-sound word pair table. Each module corresponds to the processing of the relevant part in the aforementioned method, and its internal details will not be repeated. The aforementioned modules can be implemented by loading the corresponding program on one or more processors running on the scene side, or the storage and updating of the semantic association network can be deployed locally or on a computing device that collaborates with the scene side.

[0074] In one embodiment, the acquisition module includes a sound pickup device, an image acquisition device, and an electronic tag reader; the immersive presentation module includes scene-side display, audio, and physical object presentation devices; and the remaining modules are implemented by a processor performing corresponding processing. Among the aforementioned processing steps, word-by-word recognition, code-switching point localization and intersection calculation of associated neighborhoods, as well as voice distance measurement and common associated node count statistics, are all lightweight computations. The batch construction of the near-phonetic word pair table is performed offline or incrementally as described above and is not included in the real-time rhythm of acquisition and presentation. The single computational load of each of the aforementioned processing steps is within the processing capacity of a conventional edge processor and can be completed within the real-time rhythm of acquisition and presentation without additional acceleration hardware. Therefore, the embodiments of this application can run the aforementioned closed loop in real time on conventional computing power on the scene side.

[0075] In the foregoing embodiments, the set time window and set threshold for co-occurrence, the set time period for the response duration baseline, and the shortened reproduction time interval are all given as value ranges. Similarly, the set proximity threshold and set node number threshold for fake friend determination, as well as the number of settings involved in native language regression and cancellation determination, are also given as value ranges. Typical values ​​within these ranges are merely preferred embodiments and not limitations on the values. In one embodiment, the aforementioned parameters are started with default typical values ​​upon initial installation. Subsequently, based on actual statistics of the child's speech rate, response speed, and scene switching, they are adaptively tuned within their respective ranges. For example, for children with slower speech rates, the larger side of the set time window range is used; for children with larger response fluctuations, the larger side of the set time period range is used. Thus, each parameter has both a clear value range and a typical value, and can be adjusted within the range according to individual differences. Those skilled in the art can implement this without being limited to a specific value.

[0076] This application's embodiments are applicable to situations where young children spontaneously generate mixed language discourse in an immersive bilingual environment, and their native language and target language can be distinguished by acoustic features. If a child rarely performs code-switching within a certain period, the code-switching points are sparse, and the updating of bidirectional semantic signals slows down accordingly. In this case, the system can still maintain the semantic association network based on the co-occurrence of bilingual tags when the referred object is present, and present the target language content in a targeted manner based on the gap nodes, so that the function is not interrupted. All the aforementioned processing takes multimodal acquisition of immersive scenarios as input, the construction, updating, and querying of semantic association networks as the core, and the presentation of target language content as the output, constituting a complete technical process from language data acquisition to semantic association calculation and then to content presentation.

[0077] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An immersive bilingual semantic association system for preschoolers, characterized in that, include: The data acquisition module is used to acquire children's voices in the immersive scene, images and objects in the immersive scene, and children's pointing or grasping actions. The semantic association network construction module is used to establish a pair of related nodes between mother language tags and target language tags whose co-occurrence frequency reaches a set threshold in the individualized cross-lingual semantic association network (hereinafter referred to as semantic association network) based on the number of times the mother language tag and the target language tag co-occur when the same referent is present. The code-switching detection module is used to identify code-switching points between the native language and the target language in children's mixed language discourse; The bidirectional semantic signal module is used to mark the node corresponding to the word spoken in the target language at the code-switching point in the semantic association network as an activated node, and to mark the node corresponding to the concept expressed by the child in the native language at the code-switching point in the target language as a gap node. The immersive presentation module is used to select, based on the distribution of activated nodes and gap nodes in the semantic association network, the target language content associated with the activated nodes or the target language content corresponding to the gap nodes as the content to be presented in the immersive scene in the next step.

2. The immersive bilingual semantic association system for preschoolers according to claim 1, characterized in that, The semantic association network construction module is also used to: determine the physical objects in the immersive scene through image recognition, or read the electronic tags on the physical objects to determine the presence of the object being referred to; A co-occurrence of the native language label and the target language label around the referent is defined as a single occurrence of the native language label and the target language label around the referent, provided that the native language label and the target language label appear sequentially during the presence of the same referent and the time interval between their occurrences does not exceed a set time window. When the number of times the native language tag and the target language tag co-occurs reaches a set threshold, the two are established as a pair of related nodes in the individualized cross-linguistic semantic association network. When the association determined by the co-occurrence frequency is inconsistent with the translation relationship given by the pre-set bilingual dictionary, the semantic association network is updated with the association determined by the co-occurrence frequency.

3. The immersive bilingual semantic association system for preschoolers according to claim 1, characterized in that, The code-switching detection module is also used to: use a language recognition model to label the language to which each word in the child's mixed language speech belongs; The position where the language of two adjacent words changes from the mother tongue to the target language, or vice versa, is determined as the code-switching point; For the word spoken in the target language at the code-switching point, extract the context words in the native language discourse before and after the code-switching point that point together refer to the same referent as the word spoken in the target language. Use the context words to determine the semantic node corresponding to the word spoken in the target language in the semantic association network. The semantic node is marked as an activated node by the bidirectional semantic signal module.

4. The immersive bilingual semantic association system for preschoolers according to claim 3, characterized in that, The training process of the language recognition model includes: using the acoustic features of sample words as input to the language recognition model; The output of the language identification model is used to determine the language to which the sample word belongs; The language identification model is trained using the language label of the sample word as the training label until the difference between the output and the language label converges.

5. The immersive bilingual semantic association system for preschoolers according to claim 1, characterized in that, The immersive presentation module is also used to: select the target language content corresponding to the gap node in the target language for the concept marked as a gap node, and use it as the content to be presented in the immersive scene in the next step; For concepts labeled as activated nodes, determine the nodes to be presented that are associated with the activated nodes in the semantic association network; Nodes in the nodes to be presented that are associated with both the native language tag and the target language tag are identified as shared associated nodes, and nodes in the nodes to be presented that are associated only with the target language tag are identified as exclusive associated nodes. First, the target language content corresponding to the shared associated node is presented, and then the target language content corresponding to the proprietary associated node is presented.

6. The immersive bilingual semantic association system for preschoolers according to claim 1, characterized in that, The immersive presentation module is also used to: record the response time of the child from the end of the presentation of the target language content to the child's correct response to the target language content; The rolling average of the child's multiple response times within a set time period is determined as the baseline for the child's response time. Nodes corresponding to target content with response times longer than the baseline response time will be retained in the presentation rotation of the immersive scene, and the reproduction time interval of the nodes will be shortened. Nodes corresponding to target content whose response time is not higher than the baseline response time are removed from the presentation rotation.

7. The immersive bilingual semantic association system for preschoolers according to claim 1, characterized in that, It also includes a native language regression monitoring module, which is used to: record the immersive scene that leads to the target language concept for the target language concept that has been marked as an activated node; In an immersive scenario that introduces the target language concept, if a child is detected speaking the target language concept in their native language, it is recorded as a native language regression event. The activated node that has experienced the native language fallback event will be immediately inserted into the head of the presentation queue of the immersive presentation module for playback, without waiting for the scheduled playback time of the activated node.

8. The immersive bilingual semantic association system for preschoolers according to claim 7, characterized in that, The native language regression monitoring module is also used to: determine that the current immersive scene and the immersive scene that evokes the target language concept are the same scene by matching the image features of the immersive scene or by determining that the same referent is present; Count the cumulative number of times the native language rollback event occurs on the same activated node; When the cumulative number of times reaches a set number, the reproduction frequency of the activated node in the presentation queue is increased; The native language rollback event is recorded on the premise that the target language concept has been previously marked as an activated node, and concepts that have not been previously marked as activated nodes are not recorded.

9. The immersive bilingual semantic association system for preschoolers according to claim 1, characterized in that, It also includes a fake friend disambiguation module, which is used to: determine the distance between the phonetic features of the target word and the phonetic features of the native word, and determine the reciprocal of the distance as the phonetic similarity between the target word and the native word; Determine the number of common association nodes between the target word and the mother word in the semantic association network; When the voice proximity is higher than a set proximity threshold and the number of common associated nodes is lower than a set number of nodes threshold, the target word and the native word are identified as a false friend risk word pair. In the semantic association network, the scene with the fewest associated nodes with the target word is identified, and the target word is forcibly paired with the scene and presented in the immersive scene.

10. The immersive bilingual semantic association system for preschoolers according to claim 9, characterized in that, The fake friend disambiguation module is also used to: traverse the target words and native words that the child has been exposed to, pair words whose phonetic similarity is higher than the set similarity threshold, and construct a cross-language phonetic similarity word pair table; The number of common associated nodes is continuously monitored for word pairs in the cross-linguistic phonetic similarity word pair table; When the target word and the native word are determined to be a false friend risk pair, and the child makes a correct response to the forced pairing presentation continuously within a set number of times, the determination of the target word and the native word as a false friend risk pair is lifted. When the number of common associated nodes of word pairs in the cross-language homophone word pair table reaches the set node number threshold, the word pair is removed from the fake friend risk word pair.

Images