Method and system for adapting neural networks to dynamic environments

By training neural networks to correlate background and foreground features, the method enhances their adaptability to dynamic environments, addressing performance issues and expanding their applicability.

JP7578779B2Active Publication Date: 2024-11-06HITACHI LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023176429
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-11-02
Filing Date
2023-10-12
Publication Date
2024-11-06
Estimated Expiration
2043-10-12

AI Technical Summary

Technical Problem

Neural networks struggle to adapt to dynamic environments, leading to performance issues due to variations in backgrounds and climatic conditions, and existing techniques like semi-supervised learning and image view extraction are inadequate for adapting without extensive data and annotations.

Method used

A method and system that trains neural networks by correlating first and second sets of features (background and foreground) using correlation pairs, allowing the network to determine the foreground features based on the correlated pairs, thereby adapting to new environments.

Benefits of technology

Enables neural networks to effectively adapt to different environments, improving performance and scalability, especially under data scarcity constraints, such as new IoT environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578779000003
    Figure 0007578779000003
  • Figure 0007578779000004
    Figure 0007578779000004
  • Figure 0007578779000005
    Figure 0007578779000005
Patent Text Reader

Abstract

To provide a method and system for adapting a neural network to a dynamic environment.SOLUTION: A method includes: receiving a first set of a plurality of training features; and determining a second set of features by associating the first set of features with one or more correlation pairs generated using a neural network. The one or more correlation pairs generates one or a plurality of first labels and one or plurality of second labels for the dataset by receiving feature sets consisting of the first set of training features and a second set of training features of dataset for each of a plurality of training data items and projecting them onto a feature space. The first set of training features and the second set of training features are rearranged on the feature space by correlating them with each other. One or a plurality of correlation pairs each representing one or a plurality of first labels and corresponding second labels of the data set are generated.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to the field of artificial intelligence and data analysis. More particularly, the present disclosure relates to a method and system for adapting neural networks to dynamic environments. [Background technology]

[0002] A neural network is a computational model that attempts to mimic the way the human brain makes decisions. Neural networks are widely used in a variety of applications across different industries. When used in such applications, neural networks adapt to different types of environments. For example, a neural network implemented in a video analytics system, such as a surveillance application for workplace safety, needs to identify workers wearing safety gear on an assembly line. Also, the neural network needs to identify workers wearing safety gear on a production line. Because the environments of an assembly line and a production line are different, there is a large amount of background variability that affects the identification of workers wearing safety gear. This can cause the neural network to misidentify or miss identifying workers, which reduces the performance of the neural network. In another example, in an object recognition application, a neural network trained using images of objects taken in a sunny area may not work well in a cloudy area. The neural network may not work well under different weather conditions because it may misidentify objects. Therefore, neural networks need to be developed for different environments and their performance improved. However, such development is expensive and tedious. Additionally, the lack of data and annotations in various applications makes it difficult to adapt neural networks to different environments.

[0003] Conventional techniques such as semi-supervised learning, zero-shot learning, and few-shot learning address the need for large amounts of data and annotations. However, these techniques do not provide a solution to extend neural networks to deal with different environments without minimal annotations. Moreover, these techniques have low accuracy despite producing annotations. Some other conventional techniques utilize image view extraction to improve the performance of neural networks. However, these conventional techniques are unable to adapt to different backgrounds and environments because they only consider foreground features in the data that are relevant to the application of the neural network.

[0004] The information disclosed in the Background section of this disclosure is intended to enhance the understanding of the general background of the present invention and should not be construed as an admission or in any way suggesting that this information forms prior art already known to those of ordinary skill in the art. Summary of the Invention

[0005] In an embodiment, the present disclosure discloses a method for adapting a neural network to a dynamic environment. The method comprises receiving a first set of features of the input data items associated with a first portion of the input data items. The method further comprises determining a second set of features of the input data items associated with a second portion of the input data items by associating the first set of features with one or more correlation pairs generated using the neural network. The one or more correlation pairs are generated by receiving a feature set including a first set of training features and a second set of training features of each data set of the plurality of training data items from the one or more neural networks. The one or more first labels and the one or more second labels for each data set of the plurality of training data items are generated by projecting the first set of training features and the second set of training features onto a feature space. The first set of training features and the second set of training features are relocated on the feature space based on the correlation between the first set of training features and the second set of training features. One or more correlation pairs, each indicating one or more first labels and corresponding second labels for each of the datasets of the plurality of training data items, are generated based on the rearrangement of the first set of training features and the second set of training features.

[0006] In one embodiment, the present disclosure discloses a system for adapting a neural network to a dynamic environment. The system includes one or more processors and a memory. The one or more processors are configured to receive a first set of features of the input data items associated with a first portion of the input data items. Furthermore, the one or more processors are configured to determine a second set of features of the input data items associated with a second portion of the input data items by associating the first set of features with one or more correlation pairs generated using the neural network. The one or more correlation pairs are generated by receiving a feature set including a first set of training features and a second set of training features of each dataset of the plurality of training data items from the one or more neural networks. The one or more first labels and the one or more second labels for each dataset of the plurality of training data items are generated by projecting the first set of training features and the second set of training features onto a feature space. The first set of training features and the second set of training features are relocated on the feature space based on the correlation between the first set of training features and the second set of training features. One or more correlation pairs, each indicating one or more first labels and corresponding second labels for each of the datasets of the plurality of training data items, are generated based on the rearrangement of the first set of training features and the second set of training features.

[0007] The foregoing summary is illustrative and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0008] The novel features and characteristics of the present disclosure are set forth in the appended claims. However, the disclosure itself, as well as its preferred modes of use, further objects and advantages, will be best understood by reference to the following detailed description of exemplary embodiments, when read in conjunction with the accompanying drawings. One or more embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which like reference numerals represent like elements throughout. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example environment for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. [Diagram 2] FIG. 2 is a detailed diagram of a system for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. [Figure 3A] FIG. 3A is an exemplary diagram for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. [Figure 3B] FIG. 3B is an exemplary diagram for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. [Figure 3C] FIG. 3C is an exemplary diagram for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. [Figure 4] FIG. 4 is an exemplary flowchart illustrating method steps for training a neural network to adapt to a dynamic environment, according to some embodiments of the present disclosure. [Diagram 5] FIG. 5 is an exemplary flowchart illustrating method steps for adapting a neural network to a dynamic environment according to some embodiments of the present disclosure. [Figure 6] FIG. 6 is a block diagram of a general-purpose computing system for adapting neural networks to dynamic environments according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Those skilled in the art will appreciate that any block diagrams herein represent conceptual views of example systems embodying the principles of the present subject matter. Similarly, any flow charts, flow diagrams, state transition diagrams, pseudocode, or the like will be understood to represent various processes that may be substantially represented on a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0011] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0012] While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are described in detail below. It is to be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0013] "Comprises," "comprising," or other variations thereof are intended to be non-exclusive inclusive, and a set-up, apparatus, or method consisting of a list of components or steps does not include only those components or steps, but may include other components or steps not expressly listed or inherent to such set-up or apparatus or method. In other words, a statement "comprising..." of one or more elements in a system or apparatus does not, absent further constraints, preclude the presence of other or additional elements in the system or apparatus.

[0014] Neural networks are used in various applications in various industries. When used in such applications, neural networks respond to different types of environments. To obtain better performance, neural networks need to adapt to different types of environments. The present disclosure provides a method and system for adapting a neural network to different dynamic environments. The neural network is trained by providing a first set of training features (e.g., background features) and a second set of training features (e.g., foreground features) of a plurality of training data items. The neural network correlates the first set of features and the second set of features to form correlation pairs with a first label (e.g., background label) and a second label (e.g., foreground label). When the neural network receives the first set of features (e.g., background features) of the input data items, the neural network determines the second set of features (e.g., foreground features) of the input data items based on the correlation pairs. Thus, the neural network can adapt to new environments. Thus, the present disclosure enables the neural network to adapt to different environments. The present disclosure can be used in developing neural networks under data scarcity constraints for new environments (e.g., new Internet of Things (IoT) environments), thereby expanding the applications of neural networks.

[0015] FIG. 1 illustrates an exemplary environment 100 for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. The environment 100 includes a neural network 102 and a system 104. An input data item 101 may be provided to the neural network 102. The input data item 101 includes one of an image, a video, an audio input, a text input, a voice input, and the like. The input data item 101 may be received from a source such as a capture unit, a voice recorder, a database, and the like. The system 104 receives a first set 103 of features of the input data item 101 (first set of features 103) associated with a first portion of the input data item 101. For example, if the input data item 101 is an image, the first portion of the input data item 101 may be a background of the image. In one embodiment, the first set of features 103 may be received from the neural network 102. The neural network 102, also known as an artificial neural network, is a computing system having interconnected nodes. The neural network 102 recognizes hidden patterns and correlations in data, domains, and classifications over time for continuous learning and improvement. The neural network 102 is composed of at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), a recurrent neural network, a graph convolutional network, a sequential neural network, an encoder-decoder network, and combinations thereof, etc. Those skilled in the art will understand that the present disclosure is applicable to any neural network other than the above neural networks.

[0016] The system 104 may determine a second set 108 of features of the input data item 101 (second set of features 108) associated with a second portion of the input data item 101 by associating the first set 103 of features with one or more correlation pairs. For example, if the input data item 101 is an image, the second portion of the input data item 101 may be the foreground of the image. As used herein, the foreground of the input data item 101 refers to a portion of the input data item 101 that includes objects of interest / audio features (if the input data item 101 includes an audio input) that are relevant to a given application of the neural network 102. The background of the input data item 101 refers to a portion of the input data item 101 / audio features that do not include objects of interest / are not relevant to a given application of the neural network 102. The correlation pairs are generated by receiving a feature set consisting of a first set of training features (first set of training features) and a second set of training features (second set of training features) of each data set of a plurality of training data items using the neural network 102. For example, the first set of training features may be background training features and the second set of training features may be foreground training features. Each dataset of the plurality of training data items may be composed of an original data item and one or more transformations of the original data item. The first set of training features and the second set of training features are projected onto a feature space to generate a first label and a second label. Furthermore, the first set of training features and the second set of training features are rearranged onto the feature space by correlating the first set of training features with the second set of training features. Based on the rearrangement, one or more correlation pairs are generated, each indicating a first label and a corresponding second label. The correlation pairs indicate a first label (e.g., a background label) and a corresponding second label (e.g., a foreground label). When a new input data item 101 having new features (e.g., a background feature) is received, the neural network 102 can determine a foreground feature of the input data item 101 and adapt to a different environment. Neural networks 102 can be implemented in applications such as object detection, object tracking, and face recognition.

[0017] The system 104 may include an input / output (I / O) interface 106, a memory 107, and a central processing unit 105 (also referred to as a "CPU" or "one or more processors 105"). In some embodiments, the memory 107 may be communicatively coupled to the one or more processors 105. The memory 107 stores instructions executable by the one or more processors 105. The one or more processors 105 may include at least one data processor for executing program components for executing user or system generated requests. The memory 107 may be communicatively coupled to the one or more processors 105. The memory 107 stores instructions executable by the one or more processors 105 that, when executed, cause the one or more processors 105 to adapt the neural network 102 to a dynamic environment. The I / O interface 106 is coupled to the one or more processors 105 to which input signals or / and output signals are communicated. For example, an input data item 101 may be received from a user via the I / O interface 106. In one embodiment, the system 104 may be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a personal computer (PC), a notebook, a smartphone, a tablet, a server, a network server, a cloud-based server, and the like.

[0018] A detailed diagram 200 of a system 104 for adapting a neural network 102 to different environments is shown, according to some embodiments of the present disclosure. In one embodiment, the memory 107 may include one or more modules 202 and data 201. The one or more modules 202 may be configured to perform steps of the present disclosure using the data 201 to adapt the neural network 102 to a dynamic environment. In one embodiment, each of the one or more modules 202 may be a hardware unit that is external to the memory 107 and may be coupled with the system 104. As used herein, the term module 202 refers to an Application Specific Integrated Circuit (ASIC), an electronic circuit, a Field-Programmable Gate Array (FPGA), a Programmable System-on-Chip (PSoC), a combinational logic circuit, and / or other suitable components that provide the described functionality. The one or more modules 202, when configured with the described functionality defined in this disclosure, result in novel hardware.

[0019] In one embodiment, modules 202 can include, for example, input module 209, label generation module 210, rearrangement module 211, correlation module 212, decision module 213, and other module 214. It will be appreciated that such aforementioned modules 202 may be represented as a single module or a combination of different modules. In one implementation, data 201 can include, for example, input data 203, label data 204, rearrangement data 205, correlation data 206, decision data 207, and other data 208.

[0020] In one embodiment, the input module 209 may be configured to receive a feature set for a dataset of each of the plurality of training data items. The plurality of training data items may include one of an image, a video, an audio input, a text input, a voice input, and the like. Each dataset of the plurality of training data items is comprised of an original data item and one or more transformations of the original data item. The one or more transformations of the original data item may comprise at least one of a rotation, a translation, a scaling, and the like. For example, the dataset may be comprised of an original image, a rotated version of the original image, a scaled version of the original image, and the like. The feature set may be comprised of a first set of training features and a second set of training features for each dataset of the plurality of training data items. The first set of training features may be associated with a first portion of each of the plurality of training data items. The second set of training features may be associated with a second portion of each of the plurality of training data items. For example, consider the training data items to be images. In that case, the first portion of the training data items may be the background of the image. Meanwhile, the second portion of the training data items may be the foreground of the image. The image consists of a train in the foreground and people in a train station in the background. In another example, the image consists of workers in the foreground and an assembly line environment in the background.

[0021] In one embodiment, during training of the neural network 102, the input module 209 may receive a first set of training features and a second set of training features from one or more neural networks 102. For example, the first neural network may be used to extract a first set of training features associated with a first portion of a data set of each of the multiple training data items. The second neural network may be used to extract a second set of training features associated with a second portion of a data set of each of the multiple training data items. The one or more neural networks 102 may be trained neural networks or untrained neural networks. In one embodiment, the multiple training data items may be provided to the input module 209. The input module 209 may generate the first and second portions of each of the multiple training data items. For example, the input module 209 may generate the first and second portions of each of the multiple training data items using a background subtraction technique to obtain a foreground and background of each of the multiple training data items. Those skilled in the art will appreciate that any technique other than the techniques described above may be used to obtain the first and second portions of each of the plurality of training data items. In another embodiment, the input module 209 may receive the first and second portions of each of the plurality of training data items. The first and second portions may be received from a user, a database, etc.

[0022] Referring to the example 300 of FIG. 3A, a first portion 302 and a second portion 301 of each of the plurality of training data items are illustrated. In this example, each of the plurality of training data items is an image. The first portion 302 is the background of the image, and the second portion 301 is the foreground of the image. Furthermore, one or more transformations 303 of each of the plurality of training data items are generated. Then, one or more neural networks 102 (neural network 1021, neural network 1022) can extract a first set of features and a second set of features of the dataset of each of the plurality of training data items. For example, the neural network 1021 generates the second set of features 304 (e.g., foreground features), and the neural network 1022 generates the first set of features 305 (e.g., background features). In one example, the neural network 102 may be trained for multiple classes of each of the plurality of training data items based on tasks performed by the neural network 102 in different applications. A first set of features 305 and a second set of features 304 corresponding to each class may be extracted. The feature sets may be stored in memory 107 as input data 203.

[0023] In one embodiment, the label generation module 210 may be configured to receive the input data 203 from the input module 209. Furthermore, the label generation module 210 may be configured to generate one or more first labels and one or more second labels for each data set of the plurality of training data items. The label generation module 210 generates one or more first labels (e.g., background labels) and one or more second labels (e.g., foreground labels) by projecting the first set of training features and the second set of training features onto a feature space. First, the label generation module 210 projects the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the training data items from the plurality of training data items. For example, the training data items may be images. The one or more classes of the second portion of the image (e.g., foreground) may include object identification, object recognition, etc. In one example, the neural network 102 may be implemented in pedestrian classification. In that case, the one or more classes may include pedestrian identification, pedestrian recognition, etc. In another example, the neural network 102 may be implemented in a monitoring application for workplace safety. The one or more classes may include worker identification, safety equipment compliance detection, etc. In one embodiment, the feature space may be a hypersphere. Those skilled in the art will appreciate that the first and second sets of training features may be projected into any feature space other than the feature spaces mentioned above. Referring to 306 in FIG. 3B, the triangles indicate training features from the second set of training features projected into the feature space 308. Similarly, the stars indicate training features from the first set of training features projected into the feature space 308. The label generation module 210 then creates multiple regions based on the similarity between the first set of training features and the second set of training features. Referring to the example of FIG. 3B, multiple regions 307 11 , 307 12 , 307 21 , 307 22 , 307 31 and 307 32is created. If the feature space is a hypersphere, the multiple regions are spheres. In other examples, the multiple regions may include squares, eclipses, etc. 309 shows a curved hyperplane 309 that separates the first set of training features and the second set of training features.

[0024] The label generation module 210 may identify a center and a radius for each of the multiple regions. The label generation module 210 then adjusts the radius for each of the multiple regions by projecting the first set of training features and the second set of training features in a feature space for each of the one or more classes of the second portion of the other training data items from the multiple training data items. If the feature space is a hypersphere, the label generation module 210 adjusts the radius based on the following equation (1):

[0025]

number

[0026] In one embodiment, the rearrangement module 211 may be configured to receive input data 203 from the input module 209 and label data 204 from the label generation module 210. Furthermore, the rearrangement module 211 may be configured to rearrange the first set of training features and the second set of training features on the feature space based on the correlation between the first set of training features and the second set of training features. In particular, the rearrangement module 211 correlates the one or more second labels with the one or more first labels by providing the one or more first labels to the first neural network from the one or more neural networks 102 associated with the generation of the second set of training features. Furthermore, the one or more second labels are provided to the second neural network from the one or more neural networks 102 associated with the generation of the first set of training features. The correlation may be performed using a swapped labeling technique. Those skilled in the art will understand that any technique other than the above-mentioned techniques may be used to correlate the one or more second labels with the one or more first labels. The relocation module 211 may relocate the first set of training features and the second set of training features in the feature space based on the correlation. In one example, the relocation may include moving a region having the first set of training features relative to a region having the second set of training features. Referring to the example of FIG. 3B, based on the correlation, region 307 12 Area 307 11 Data relating to the repositioned first set of training features and the second set of training features may be stored in memory 107 as repositioned data 205.

[0027] In one embodiment, the correlation module 212 may be configured to receive the rearrangement data 205 from the rearrangement module 211. Additionally, the correlation module 212 may be configured to generate one or more correlation pairs. The correlation module 212 may generate the one or more correlation pairs based on the rearrangement of the first set of training features and the second set of training features. Each of the one or more correlation pairs may indicate one or more first labels and corresponding second labels of each of the data sets of the multiple training data items. Referring to the example of FIG. 3B, the first correlation pair may indicate the region 307 11 and area 307 12 and the second correlation pair includes region 307 21 and area 307 22 and the third correlation pair includes region 307 31 and area 307 32 In one example, the correlation pairs may indicate a background label and a corresponding foreground label for each of a data set of multiple images. One or more correlation pairs may be stored in memory 107 as correlation data 206.

[0028] In one embodiment, during deployment, the input module 209 is further configured to receive a first set of features 103 of the input data item 101. The input data item 101 may include one of an image, a video, an audio input, a text input, and an audio input, and the like. The first set of features 103 may be associated with a first portion of the input data item 101. For example, the input data item 101 may be an image. The first portion of the input data item 101 may be a background of the image. In one example, the background of the image may include a train in an open area. In another example, the input data item 101 may be written text. The background may include blue paper. In another example, the background includes a production line environment. The neural network may be trained in an assembly line environment. In one embodiment, the input module 209 may receive the first set of features 103 from the neural network 102. In another embodiment, the input module 209 may receive the first set of features from one or more sources, such as a database, a user, a capture unit, an audio recorder, and the like. The first set of features 103 may be stored in the memory 107 as input data 203 .

[0029] In one embodiment, the determination module 213 is configured to receive the input data 203 from the input module 209. Furthermore, the determination module 213 is configured to determine a second set of features 108 of the input data item 101. The second set of features 108 is associated with a second portion of the input data item 101, e.g., the foreground of the image. The determination module 213 may determine the second set of features 108 by associating the first set of features 103 with one or more correlation pairs. The determination module 213 may associate the first set of features 103 with one or more correlation pairs by generating a transformation matrix Wt 310, as shown in FIG. 3C. In one embodiment, the transformation matrix Wt may be generated using a Riemannian conditioning technique. The determination module 213 may associate the first set of features 103 with one or more correlation pairs using equation (2), as given below:

[0030]

number

[0031] Other data 208 may store data including temporary data and temporary files generated by one or more modules 202 to perform various functions of system 104. Other data 208 may be stored in memory 107. One or more modules 202 may also include other modules 214 to perform various miscellaneous functions of system 104. For example, other module 214 may include a test module. The test module may be configured to generate one or more test features based on the second set of training features and the second set of features 108. Furthermore, the test module may be configured to identify deviations by comparing the one or more test features with the first set of features 103. The test module identifies deviations to determine whether a correct feature (background feature) associated with the first portion of the input data item 101 is predicted. The test module may then be configured to modify the one or more test features in response to identifying the deviations. Furthermore, the test module may be configured to modify the second set of features 108. The test module is used to perform an inverse transformation to modify the second set of features 108. It will be appreciated that one or more of the modules 202 may be represented as a single module or a combination of different modules.

[0032] 4 is an exemplary flowchart illustrating method steps for training a neural network 102 to adapt to a dynamic environment, according to some embodiments of the present disclosure. As shown in FIG. 4, the method 400 may be comprised of one or more steps. The method 400 may be described in the general context of computer-executable instructions. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, modules, and functions that perform particular functions or implement particular abstract data types.

[0033] The order in which method 400 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to perform the method. Additionally, individual blocks can be deleted from the method without departing from the scope of the subject matter described herein. Additionally, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

[0034] In step 401, the system 104 receives a first set of features 103 of an input data item 101. The input data item 101 may consist of one of an image, a video, an audio input, a text input, a voice input, and the like. The first set of features 103 is associated with a first portion of the input data item 101. In one embodiment, the system 104 may receive the first set of features 103 from a neural network 102. In another embodiment, the system 104 may receive the first set of features from one or more sources, such as a database, a user, a capture unit, an audio recorder, and the like.

[0035] In step 402, the system 104 generates one or more first labels and one or more second labels for each data set of the plurality of training data items. The system 104 generates the one or more first labels and the one or more second labels by projecting the first set of training features and the second set of training features onto a feature space. The system 104 projects the first set of training features and the second set of training features in the feature space for each of the one or more classes of the second portion of the training data items from the plurality of training data items. Furthermore, the system 104 creates a plurality of regions based on the similarity between the first set of training features and the second set of training features. The system 104 then identifies a center and a radius for each of the plurality of regions. The system 104 adjusts the radius for each of the plurality of regions by projecting the first set of training features and the second set of training features into the feature space for each of the one or more classes of the second portion of the other training data items from the plurality of training data items. Additionally, the system 104 generates a label for each of the multiple regions associated with the corresponding region based on a similarity between the first set of training features and the second set of training features.

[0036] In step 403, the system 104 rearranges the first set of training features and the second set of training features on the feature space based on the correlation between the first set of training features and the second set of training features. The system 104 correlates the one or more second labels with the one or more first labels by providing the one or more first labels to the first neural network from the one or more neural networks 102 associated with the generation of the second set of training features. Additionally, the one or more second labels are provided to the second neural network from the one or more neural networks 102 associated with the generation of the first set of training features.

[0037] In step 404, the system 104 generates one or more correlation pairs. The system 104 may generate the one or more correlation pairs based on rearrangement of the first set of training features and the second set of training features. Each of the one or more correlation pairs may indicate one or more first labels and corresponding second labels of each of the data sets of the multiple training data items.

[0038] 5 is an exemplary flowchart illustrating method steps for adapting a neural network 102 to a dynamic environment, according to some embodiments of the present disclosure. As shown in FIG. 5, the method 500 may be comprised of one or more steps. The method 500 may be described in the general context of computer-executable instructions. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, modules, and functions that perform particular functions or implement particular abstract data types.

[0039] The order in which method 500 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to perform the method. Additionally, individual blocks can be deleted from the method without departing from the scope of the subject matter described herein. Additionally, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

[0040] In step 501, the system 104 receives a first set of features 103 of an input data item 101. The input data item 101 may consist of one of an image, a video, an audio input, a text input, a voice input, and the like. The first set of features 103 is associated with a first portion of the input data item 101. In one embodiment, the system 104 may receive the first set of features 103 from a neural network 102. In another embodiment, the system 104 may receive the first set of features from one or more sources, such as a database, a user, a capture unit, an audio recorder, and the like.

[0041] In step 502, the system 104 determines a second set 108 of features of the input data item 101. The second set 108 of features is associated with a second portion of the input data item 101. The system 104 determines the second set 108 of features by associating the first set 103 of features with one or more correlation pairs. The system 104 may associate the first set 103 of features with one or more correlation pairs by generating a transformation matrix Wt using a Riemannian conditioning technique. Those skilled in the art will understand that any technique other than the above-mentioned techniques may be used to associate the first set 103 of features with one or more correlation pairs.

[0042] The above-mentioned embodiments and corresponding figures are described by considering the input data items (the plurality of training data items and the input data item 101) as an image. Thus, the first set of features, the second set of features, the first set of training features, the second set of training features, the first label, the second label, etc. correspond to the foreground and background of the image. The present invention is not limited to images, and the first set of features, the second set of features may vary accordingly based on the type of the input data item 101. For example, the input data item 101 may include a speech input for a neural network speech recognition application. In such a case, the first set of features may consist of background features of the speech input, such as the person's accent. The second set of features may consist of foreground features of the speech input, such as the person's voice. In another example, the input data item 101 may include a speech input in a neural network accent classification application. In such a case, the first set of features may consist of background features of the speech input, such as the person's voice. The second set of features may consist of foreground features of the speech input, such as the person's accent.

[0043] <Computer System> 6 is a block diagram of an exemplary computer system 600 for implementing embodiments consistent with the present disclosure. In an embodiment, the computer system 600 may be the system 104. Thus, the computer system 600 may be used to adapt the neural network 102 in a dynamic environment. The computer system 600 may include a central processing unit 602 (also referred to as a "CPU" or "processor"). The processor 602 may include at least one data processor. The processor 602 may include specialized processing units such as an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, etc. The processor 602 may be used to implement the processor 105 described in FIG. 2.

[0044] The processor 602 may be disposed in communication with one or more input / output (I / O) devices (not shown) via an I / O interface 601. The I / O interface 601 may include, but is not limited to, communication protocols / methods such as audio, analog, digital, mono, RCA, stereo, IEEE (Institute of Electrical and Electronics Engineers)-1394, serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, digital visual interface (DVI), high definition multimedia interface (HDMI), radio frequency (RF) antenna, S-video, VGA, IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., code division multiple access (CDMA), high speed packet access (HSPA+), Global System for Mobile Communications (GSM), Long Term Evolution (LTE), WiMax, etc.), and the like.

[0045] Using the I / O interface 601, the computer system 600 may communicate with one or more I / O devices. For example, the input device 610 may be an antenna, a keyboard, a mouse, a joystick, an (infrared) remote control, a camera, a card reader, a fax machine, a dongle, a biometric reader, a microphone, a touch screen, a touch pad, a trackball, a stylus, a scanner, a storage device, a transceiver, a video device / source, etc. The output device 611 may be a printer, a fax machine, a video display (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), a plasma, a plasma display panel (PDP), an organic light emitting diode display (OLED), etc.), an audio speaker, etc.

[0046] The processor 602 may be disposed in communication with a communication network 609 via a network interface 603. The network interface 603 may communicate with the communication network 609. The network interface 603 may employ a connection protocol including, but not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 base-T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc. The communication network 609 may include, but is not limited to, a direct interconnect, a local area network (LAN), a wide area network (WAN), a wireless network (e.g., using wireless application protocol), the Internet, etc. The network interface 603 may employ, but is not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 base-T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc. as a connection protocol.

[0047] The communication network 609 may include, but is not limited to, a direct interconnection, an electronic commerce network, a peer-to-peer (P2P) network, a local area network (LAN), a wide area network (WAN), a wireless network (e.g., using a wireless application protocol), the Internet, Wi-Fi, and the like. The first network and the second network may be a dedicated network or a shared network, which represents an association of different types of networks that communicate with each other using various protocols, such as, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), Wireless Application Protocol (WAP), and the like. Furthermore, the first network and the second network may include various network devices, including routers, bridges, servers, computing devices, storage devices, and the like. The computer system 600 may receive data items 612 (an input data item and a plurality of training data items) over the communication network 609.

[0048] In some embodiments, the processor 602 may be disposed in communication with memory 605 (e.g., RAM, ROM, etc., not shown in FIG. 6) via a storage interface 604. The storage interface 604 may connect to memory 605 including, but not limited to, memory drives, removable disk drives, etc., employing connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), Fibre Channel, Small Computer System Interface (SCSI), etc. The memory drives may further include drum, magnetic disk drives, magneto-optical drives, optical drives, Redundant Array of Independent Discs (RAID), solid state memory devices, solid state drives, etc.

[0049] The memory 605 may store a collection of program or database components, including, but not limited to, a user interface 606, an operating system 607, a web browser 608, and the like. In some embodiments, the computer system 600 may store user / application data, such as data, variables, records, and the like, as described in this disclosure. Such databases may be implemented as fault-tolerant, relational, scalable, secure databases, such as Oracle® or Sybase®. The memory 605 may be used to implement the memory 107 described in FIG. 2. The memory 605 may be communicatively coupled to the processor 602. The memory 605 may store instructions executable by one or more processors 602, which, when executed, may enable the processor 602 to adapt the neural network 102 to a dynamic environment.

[0050] Operating system 607 can facilitate resource management and operation of computer system 600. Examples of operating systems include, but are not limited to, APPLE MACINTOSH OS X, UNIX®, UNIX-like system distributions (e.g., BERKELEY SOFTWARE DISTRIBUTION™ (BSD), FREEBSD™, NETBSD™, OPENBSD™, etc.), LINUX DISTRIBUTIONS™ (e.g., RED HAT™, UBUNTU™, KUBUNTU™, etc.), IBM™ OS / 2, MICROSOFT™ WINDOWS™ (XP™, VISTA™ / 7 / 8, 10, etc.), APPLER IOS™, GOOGLER ANDROID™, BLACKBERRY® OS, etc.

[0051] In some embodiments, computer system 600 may implement program components stored in a web browser 608. Web browser 608 may be, for example, a hypertext browsing application such as MICROSOFT INTERNET EXPLORERTM, GOOGLER CHROMETM0, MOZILLAR FIREFOXTM, APPLER SAFARITM, etc. Secure web browsing is provided using Secure Hypertext Transport Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), etc. Web browser 608 may utilize facilities such as AJAXTM, DHTMLTM, ADOBER FLASHTM, JAVASCRIPTTM, JAVATM, application programming interfaces (APIs), etc. In some embodiments, computer system 600 may implement program components stored in a mail server (not shown). The mail server may be an Internet mail server such as Microsoft Exchange. The mail server may utilize facilities such as ASP™, ACTIVEXTM, ANSI™ C++ / C#, MICROSOFT®, .NET™, CGI SCRIPTS™, JAVA™, JAVASCRIPT™, PERLTM, PHP™, PYTHON™, WEBOBJECTS™, etc. The mail server may utilize communications protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFT® exchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP), etc. In some embodiments, computer system 600 may implement a mail client stored program component.The mail client (not shown) may be a mail viewing application such as APPLER MAIL™, MICROSOFT ENTOURAGE™, MICROSOFT OUTLOOK™, MOZILLAR THUNDERBIRD™, or the like.

[0052] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory in which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible objects and exclude carrier waves and transient signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, compact disk read-only memory (CD ROM), digital video disk (DVD), flash drives, disks, and any other known physical storage medium.

[0053] The present disclosure provides a method and system for adapting a neural network to different dynamic environments. The neural network is trained by providing a first set of training features (e.g., background features) and a second set of training features (e.g., foreground features) of a plurality of training data items. The neural network correlates the first set of features and the second set of features to form a correlation pair with a first label (e.g., background label) and a second label (e.g., foreground label). When the neural network receives the first set of features (e.g., background features) of the input data item, the neural network determines the second set of features (e.g., foreground features) of the input data item based on the correlation pair. Thus, the network can adapt to new environments. Thus, the present disclosure enables the neural network to adapt to different environments. The present disclosure can be used to develop neural networks under data scarcity constraints for new environments (e.g., new Internet of Things (IoT) environments), thereby expanding the applications of neural networks.

[0054] The terms "embodiment," "embodiment," "embodiment," "one or more embodiments," "some embodiments," and "one embodiment" mean "one or more (but not all) embodiments of the invention" unless expressly specified otherwise.

[0055] The terms "including," "consisting of," "having," and variations thereof mean "including, but not limited to," unless expressly stated otherwise.

[0056] A list of items does not imply that any or all of the items are mutually exclusive unless expressly stated otherwise. The terms "a," "an," and "the" mean "one or more" unless expressly stated otherwise.

[0057] A description of an embodiment having multiple components in communication with each other does not imply that all such components are required, but rather various optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0058] Where a single device or article is described herein, it will be readily apparent that multiple devices / articles (whether they cooperate or not) may be used in place of the single device / article. Similarly, where two or more devices or articles are described herein (whether they cooperate or not), it will be readily apparent that a single device / article may be used in place of the two or more devices or articles, or that a different number of devices / articles may be used in place of the number of devices or programs shown. The functionality and / or features of a device may alternatively be embodied by one or more other devices not explicitly described as having such functionality / features. Thus, other embodiments of the invention need not include the device itself.

[0059] The illustrated operations of Figures 4 and 5 show certain events occurring in a particular order. In alternative embodiments, certain operations may be performed in a different order, modified, or removed. Additionally, steps may be added to the logic described above and still be consistent with the described embodiments. Additionally, operations described herein may be performed sequentially or certain operations may be processed in parallel. Additionally, operations may be performed by a single processing unit or by distributed processing units.

[0060] Finally, the language used herein has been selected primarily for purposes of readability and explanation, and may not have been selected to define or encompass inventive subject matter. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but rather by the claims that will be issued on an application based hereon. Accordingly, the disclosure of embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is defined in the following claims.

[0061] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those of skill in the art. The various aspects and embodiments disclosed herein are intended to be illustrative and not limiting, with the true scope being indicated by the following claims. [Explanation of symbols]

[0062] 100... environment, 101... input data item, 102... neural network, 103... first set of features, 104... system, 105... processor, 106... I / O interface, 107... memory, 108... second set of features, 200... detailed view, 201... data, 202... module, 203... input data, 204... label data, 205... rearrangement data, 206... correlation data, 207... decision data, 208... other data, 209... input module, 210... label generation module, 211... rearrangement module, 212... correlation module, 213... decision module, 214... other modules , 300...example, 301...first portion, 302...second portion, 303...transformation, 304...second set of features, 305...first set of features, 307...region, 308...feature space, 309...curved hyperplane, 310...transformation matrix, 600...computer system, 601...I / O interface, 602...processor, 603...network interface, 604...storage interface, 605...memory, 606...user interface, 607...operating system, 608...web browser, 609...communication network, 610...input device, 611...output device, 612...data item

Claims

1. 1. A method for adapting a neural network to a dynamic environment, comprising: receiving, by the system, a first set of characteristics of the input data item associated with a first portion of the input data item; determining, by the system, a second set of features of the input data item associated with a second portion of the input data item by associating the first set of features with one or more correlation pairs generated using a neural network; receiving a feature set from the one or more neural networks, the feature set including a first set of training features and a second set of training features for each data set of a plurality of training data items; generating, for each data set of the plurality of training data items, one or more first labels and one or more second labels by projecting the first set of training features and the second set of training features onto a feature space; reordering the first set of training features and the second set of training features in a feature space based on a correlation between the first set of training features and the second set of training features; generating one or more correlation pairs, each indicative of one or more first labels and corresponding second labels for a respective data set of the plurality of training data items, based on a rearrangement of the first set of training features and the second set of training features; Including, method.

2. 10. The method of claim 1 , generating one or more test features based on the second set of training features and the second set of features; identifying deviations by comparing the one or more test features to the first set of features; modifying the second set of characteristics in response to the deviation; and Including, method.

3. 10. The method of claim 1 , the input data item and the plurality of training data items include one of an image, a video, an audio input, a text input, and a voice input; method.

4. 10. The method of claim 1 , each dataset of the plurality of training data items comprises an original data item and one or more transformations of the original data item; method.

5. 10. The method of claim 1 , Generating the one or more first labels and the one or more second labels includes: projecting, from the plurality of training data items, the first set of training features and the second set of training features in the feature space for each of one or more classes of a second portion of the training data items; creating a plurality of regions based on similarity between the first set of training features and the second set of training features; identifying a center and a radius for each of the plurality of regions; adjusting a radius of each of the plurality of regions by projecting the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the other training data items from the plurality of training data items; generating, for each of the plurality of regions, a label based on a similarity between the first set of training features and the second set of training features associated with a corresponding region; Including, method.

6. 6. The method of claim 5, the number of regions is based on a number of one or more transformations of each of the plurality of training data items and a number of the one or more classes. method.

7. 10. The method of claim 1 , Correlating the one or more second labels with the one or more first labels includes providing the one or more first labels to a first neural network from one or more neural networks associated with generating a second set of training features; providing the one or more second labels to a second neural network from one or more neural networks associated with generating the first set of training features; Including, method.

8. 1. A system for adapting a neural network to a dynamic environment, comprising: one or more processors; a memory storing processor-executable instructions that, when executed, cause one or more processors to perform the following operations: Equipped with The processor, Receiving a first set of characteristics of an input data item; determining a second set of features of the input data items by correlating the first set of features with one or more correlation pairs generated using a neural network; receiving a feature set from the one or more neural networks, the feature set including a first set of training features and a second set of training features for each data set of a plurality of training data items; generating, for each data set of the plurality of training data items, one or more first labels and one or more second labels by projecting the first set of training features and the second set of training features onto a feature space; repositioning the first set of training features and the second set of training features in a feature space based on a correlation between the first set of training features and the second set of training features; generating one or more correlation pairs indicative of one or more first labels and corresponding second labels for each data set of the plurality of training data items based on a rearrangement of the first set of training features and the second set of training features; To carry out system.

9. 9. The system of claim 8, The one or more processors: generating one or more test features based on the second set of training features and the second set of features; identifying deviations by comparing the one or more test features to the first set of features; modifying the second set of characteristics in response to the deviation; and To carry out system.

10. 9. The system of claim 8, The input data item and the plurality of training data items comprise one of an image, a video, an audio input, a text input, and a voice input.

11. 9. The system of claim 8, each dataset of the plurality of training data items comprises an original data item and one or more transformations of the original data item; system.

12. 9. The system of claim 8, The one or more processors: projecting, from the plurality of training data items, the first set of training features and the second set of training features in the feature space for each of one or more classes of a second portion of the training data items; creating a plurality of regions based on similarity between the first set of training features and the second set of training features; identifying a center and a radius for each of the plurality of regions; adjusting a radius of each of the plurality of regions by projecting the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the other training data items from the plurality of training data items; generating, for each of the plurality of regions, a label based on a similarity between the first set of training features and the second set of training features associated with a corresponding region; Due to generating the one or more first labels and the one or more second labels; system.

13. 13. The system of claim 12, The number of the plurality of regions is based on a number of one or more transformations of each of the plurality of training data items and a number of the one or more classes.

14. 9. The system of claim 8, The one or more processors: providing the one or more first labels from one or more neural networks associated with generating the second set of training features to a first neural network, and providing the one or more second labels from the one or more neural networks associated with generating the first set of training features to a second neural network, thereby correlating the one or more first labels with the one or more second labels. system.

Citation Information

Patent Citations

  • Target tracking method and device, electronic equipment and storage medium

    CN114255257A

  • Automated Email Assistant

    JP2022511619A