Cloud edge-based architecture and image item recognition method

Abstract

The application relates to the technical field of artificial intelligence, in particular to a cloud-edge-end-based architecture and an image object identification method, which can solve the problem of user privacy data leakage caused by sharing of neural network models of each edge layer by cloud layers to a certain extent. The cloud-edge-end-based architecture comprises a terminal device layer, an edge computing layer and a cloud computing layer, wherein the edge computing layer is configured to collect images of each terminal device, identify the collected images through a preset model, determine object information in the collected images, determine a target data set based on the object information and a sub-class label of the object information, and determine a parent model based on the target data set; and the cloud computing layer is configured to acquire each parent model and send each parent model to the edge computing layer, the application shares the parent models of each edge layer in the cloud computing layer, reduces leakage of user privacy data while sharing the parent models, and improves user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a cloud-edge-end based architecture and an image object recognition method. BACKGROUND

[0002] With the development of artificial intelligence, more and more scenarios can obtain images of objects through devices with cameras, and cloud-edge-end architecture can recognize objects through images, which brings convenience to people's life and work. In the cloud-edge-end architecture, "cloud" refers to a cloud data center, which can be referred to as a cloud layer, "edge" refers to an edge layer of the cloud, and "end" refers to a terminal device layer, including terminal devices with cameras, etc.

[0003] Taking a family scenario as an example, the cloud-edge-end architecture is used for sharing information such as objects and realizing recognition of objects. At least one edge layer in the cloud-edge-end architecture can train a neural network model through a data set corresponding to an image of a known object, and transmit the trained neural network model to the cloud layer. The cloud layer distributes the trained neural network model to the edge layer. The edge layer identifies the object included in the image according to the received neural network model and the image captured by the terminal device layer connected to the edge layer. Since the neural network model can be shared by the edge layer corresponding to each terminal device, the sharing of the neural network model can be realized.

[0004] In the training of the neural network model, there is part of user privacy data in the image captured by the user. Sharing the neural network model of each edge layer through the cloud layer can lead to leakage of user privacy data and affect user experience. SUMMARY

[0005] The present application provides a cloud-edge-end based architecture and an image object recognition method, which can solve the technical problem of leakage of user privacy data caused by sharing of the neural network model of each edge layer through the cloud layer.

[0006] Embodiments of the present application are implemented as follows:

[0007] A first aspect of an embodiment of the present application provides a cloud-edge-end based architecture, comprising:

[0008] a terminal device layer configured to obtain captured images in an environment where each terminal device is located, the captured images including objects to be identified;

[0009] The edge computing layer is in communication connection with the terminal device layer, and is configured to aggregate the collected images of the terminal devices, identify the collected images by using a preset model to determine the item information in the collected images, and determine a target data set based on the item information and a sub-class label of the item information, the target data set including at least one target image, and the sub-class label of the item included in the target image being the same as the sub-class label of the item included in the collected image, and determine a parent class model based on the target data set.

[0010] The cloud computing layer is in communication connection with the edge computing layer, and is configured to acquire the parent class models and send the parent class models to the edge computing layer.

[0011] With reference to the first aspect, in a possible implementation manner, the target data set is determined based on the item information and the sub-class label of the item information, and the edge computing layer is further configured to:

[0012] An initial data set is determined based on the item information, the sub-class label of the item information, and a parent class label corresponding to the sub-class label, the initial data set including at least one initial image;

[0013] The target data set corresponding to the initial data set is determined by using a preset generative adversarial network, and the number of images in the target data set is greater than the number of images in the initial data set.

[0014] With reference to the first aspect, in a possible implementation manner, the target data set corresponding to the initial data set is determined by using a preset generative adversarial network, and the edge computing layer is further configured to:

[0015] A similar image corresponding to the initial image with the item information is determined by using the preset generative adversarial network.

[0016] The target data set is determined based on the target image, and the target image includes the initial image and the similar image.

[0017] With reference to the first aspect, in a possible implementation manner, the edge computing layer is configured to:

[0018] The sub-class node corresponding to the item information is determined based on the item information, and the sub-class label of the sub-class node is determined.

[0019] The parent class node corresponding to the determined sub-class node is determined based on a preset knowledge data set, and a parent class label of the parent class node is determined, wherein the parent class label includes one or more sub-class labels.

[0020] With reference to the first aspect, in a possible implementation manner, the preset model includes a pre-training model and a continuous learning model, and the item information in the collected images is determined by identifying the collected images by using the preset model, and the edge computing layer is further configured to:

[0021] The pre-trained model is used to identify the collected image, determine the identifiable first item in the collected image, and determine whether the second item that cannot be identified exists in the collected image;

[0022] If the second item that cannot be identified does not exist in the collected image, the item information is determined based on the first item;

[0023] If the second item that cannot be identified exists in the collected image, the continuous learning model is determined based on the image corresponding to the second item; the identifiable third item corresponding to the second item is determined through the continuous learning model; and the item information is determined based on the first item and the third item.

[0024] In combination with the first aspect, in a possible implementation manner, the edge computing layer includes a plurality of edge nodes, and each edge node includes:

[0025] The first communication module is in communication connection with each terminal device of the terminal device layer, and the first communication module is configured to acquire the collected image and aggregate the collected images of the terminal device layer through federated learning;

[0026] The training classifier module is configured to determine the pre-trained model based on the known data set;

[0027] The continuous learning module is configured to determine the continuous learning model based on the image corresponding to the second item and the network image through federated learning;

[0028] The edge computing classifier module is in communication connection with the training classifier module, the continuous learning module, and the first communication module, and the edge computing classifier module is configured to identify the collected image based on the pre-trained model or the continuous learning model, determine the item information in the collected image, and determine the initial data set based on the item information, the sub-class label, and the parent class label of the item information; and determine the target data set corresponding to the initial data set through the preset generative adversarial network, and determine the parent class model based on the target data set.

[0029] In combination with the first aspect, in a possible implementation manner, the cloud computing layer and the plurality of edge nodes are in communication connection, and the cloud computing layer includes:

[0030] The second communication module is in communication connection with each edge node of the edge computing layer, and the second communication module is configured to acquire the parent class model of each edge node;

[0031] The parent class knowledge continuous learning module is in communication connection with the second communication module, and the parent class knowledge continuous learning module is configured to acquire the continuous learning model of each edge node;

[0032] The cloud computing total classifier module is in communication connection with the parent class knowledge continuous learning module and the second communication module, and is used for updating the model in the cloud computing total classifier module according to the parent class model of each edge node and the continuous learning model of each edge node.

[0033] A second aspect of the embodiments of the present application provides an image object identification method, applied to a cloud-edge-end based architecture, comprising:

[0034] An acquisition image is obtained, and the acquisition image includes an object to be identified;

[0035] The acquisition image is identified through a preset model to determine object information in the acquisition image;

[0036] Based on the object information and a sub-class label of the object information, a target data set is determined, and the target data set includes at least one target image, and a sub-class label of an object included in the target image is the same as a sub-class label of an object included in the acquisition image;

[0037] Based on the target data set, a parent class model is determined.

[0038] In combination with the second aspect, in a possible implementation manner, based on the object information and the sub-class label of the object information, the target data set is determined, comprising:

[0039] Based on the object information, the sub-class label of the object information and a parent class label corresponding to the sub-class label, an initial data set is determined, and the initial data set includes at least one initial image;

[0040] Through a preset generative adversarial network, a target data set corresponding to the initial data set is determined, and an image quantity of the target data set is greater than an image quantity of the initial data set.

[0041] In combination with the second aspect, in a possible implementation manner, the preset model includes a pre-training model and a continuous learning model, and the acquisition image is identified through the preset model to determine the object information in the acquisition image, comprising:

[0042] The acquisition image is identified through the pre-training model to determine a first object recognizable in the acquisition image, and it is judged whether a second object unrecognizable exists in the acquisition image;

[0043] If the second object unrecognizable does not exist in the acquisition image, the object information is determined based on the first object;

[0044] If the second object unrecognizable exists in the acquisition image, an image corresponding to the second object and a network image are determined through federated learning to determine the continuous learning model; the third image information corresponding to the second object unrecognizable is determined through the continuous learning model; and the object information is determined based on the first object and the third image information.

[0045] The beneficial effects of the present application: the cloud edge end architecture includes a terminal device layer, an edge computing layer and a cloud computing layer, wherein the terminal device layer is configured to obtain collected images in the environment where each terminal device is located, and the collected images include an article to be identified; the edge computing layer and the terminal device layer are in communication connection, the edge computing layer is configured to aggregate the collected images of each terminal device, identify the collected images through a preset model, determine the article information in the collected images; and based on the article information and the sub-class label of the article information, determine a target data set, the target data set includes at least one target image, the sub-class label of the article included in the target image and the sub-class label of the article included in the collected image are the same; and determine a parent class model based on the target data set; the cloud computing layer and the edge computing layer are in communication connection, the cloud computing layer is configured to obtain each parent class model, and send each parent class model to the edge computing layer, the present application shares each edge layer parent class model in the cloud computing layer, reduces the leakage of user privacy data while sharing the parent class model, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 A schematic diagram of a cloud edge end-based architecture in some embodiments of the present application is shown;

[0048] Figure 2 A schematic diagram of a sub-class node and a parent class node tree structure in some embodiments of the present application is shown;

[0049] Figure 3 Another schematic diagram of a cloud edge end-based architecture in some embodiments of the present application is shown;

[0050] Figure 4 A working process schematic diagram of a cloud edge end-based architecture in some embodiments of the present application is shown;

[0051] Figure 5 A flowchart of an article identification method in an image in some embodiments of the present application is shown;

[0052] Figure 6 A flowchart of determining a target data set in an article identification method in an image in some embodiments of the present application is shown;

[0053] Figure 7A flowchart illustrating a process of determining a target data set in an item recognition method in an image in some embodiments of the present application is shown. DETAILED DESCRIPTION

[0054] For the purposes of the present application, embodiments and advantages, the following description of exemplary embodiments is provided, with reference to the accompanying drawings, in which:

[0055] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0056] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0057] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0058] With the development of artificial intelligence, more and more scenarios can obtain images of items through devices with cameras, and cloud edge architecture can recognize items through images of items, bringing convenience to people's life and work.

[0059] The main role of cloud edge architecture is to share information of items, which can also be called sharing knowledge of items, where "cloud" is a cloud data center, which can be referred to as cloud layer, and in the embodiments of the present application, it can also be referred to as cloud computing layer, "edge" is the edge layer of the cloud, in the embodiments of the present application, it can also be referred to as edge computing layer, and "end" is the terminal device layer, including terminal devices with cameras and the like.

[0060] Taking an artificial intelligence application in a home scenario as an example, the cloud edge end architecture is used for sharing information such as articles, and realizing identification of the articles; at least one edge layer in the cloud edge end architecture can train a neural network model by containing a data set corresponding to an image of a known article, and transmit the trained neural network model to a cloud layer, the cloud layer distributes the trained neural network model to the edge layer, and the edge layer identifies the article included in the image according to the received neural network model and an image captured by a terminal device layer connected to the edge layer, and since the neural network model can be shared by the edge layer corresponding to each terminal device, sharing of the neural network model can be realized.

[0061] In the related art, the training of the neural network model contains an image captured by a user, and there is part of user privacy data in the image, and sharing of the neural network model of each edge layer by the cloud layer can cause leakage of the user privacy data, and affect user experience.

[0062] In some solutions of the related art, the neural network model is determined by one-time training of image data, and when a new article appears in a scenario, the neural network model cannot effectively identify the new article, and thus cannot share information of the new article to each edge layer under the same cloud layer.

[0063] To solve the above problems, the embodiment of the present application provides a cloud edge end based architecture and an image article identification method applied to the architecture, the architecture includes a terminal device layer, an edge computing layer and a cloud computing layer, wherein the terminal device layer is configured to obtain a collection image in an environment where each terminal device is located, and the collection image includes an article to be identified; the edge computing layer and the terminal device layer are in communication connection, and the edge computing layer is configured to aggregate the collection images of each terminal device, identify the collection images by a preset model, determine article information in the collection images, and determine a target data set based on the article information and a sub-class label of the article information, the target data set includes at least one target image, and a sub-class label of an article included in the target image is the same as a sub-class label of an article included in the collection image; and determine a parent class model based on the target data set; the cloud computing layer and the edge computing layer are in communication connection, and the cloud computing layer is configured to obtain each parent class model and send each parent class model to the edge computing layer, the present application shares each parent class model of the edge layer in the cloud computing layer, reduces leakage of user privacy data while sharing the parent class model, and improves user experience.

[0064] Figure 1 A schematic diagram of a cloud edge end based architecture in some embodiments of the present application is shown, as shown in Figure 1 The cloud edge end based architecture includes a terminal device layer 300, an edge computing layer 200 and a cloud computing layer 100, wherein the edge computing layer 200 and the terminal device layer 300 are in communication connection, and the cloud computing layer 100 and the edge computing layer 200 are in communication connection.

[0065] The terminal device layer 300 is configured to obtain a collection image in an environment in which each terminal device is located, and the collection image includes an article to be identified.

[0066] The terminal device of the terminal device layer includes a device integrated with various image collection devices in each application scenario, wherein the image collection device can be a visual sensor, a camera, etc., and the terminal device can be a robot of different types, for example, a robot with a robot operating system (ROS). By controlling the visual sensor, the collection image in each scenario can be collected, and the article information in the collection image can be obtained. Through communication with the edge computing layer, the collection image can be transmitted to the edge computing layer so as to transmit the article information to the edge computing layer.

[0067] It should be understood that the terminal devices can be dispersed in different positions in the scene, and the distance between the terminal devices is uncertain. The article information in the collection image obtained by the terminal devices in different geographical positions is dispersed.

[0068] The edge computing layer 200 can have multiple edge nodes, for example, as shown in Figure 1 Each edge node is configured to have the same function, and each edge node of the edge computing layer is configured to aggregate the collection images of each terminal device, identify the collection images through a preset model, determine the article information in the collection images, and determine a target data set through the article information and a sub-class label corresponding to the article information, the target data set including at least one target image, the sub-class label of the article included in the target image being the same as the sub-class label of the article included in the collection image, and determining a parent class model based on the target data set.

[0069] The edge computing layer can aggregate the collection images of each terminal device layer belonging to the same edge computing layer, and then identify the article information in the collection images through the preset model.

[0070] It should be understood that the preset model can be trained based on the training data composed of the existing images corresponding to each scene and / or a database / data set for visual object recognition, etc.

[0071] In some embodiments, the training of the preset model can train the training data through a convolutional neural network, wherein the images in the training data are normalized in the input layer, the input images are feature extracted in the convolutional layer, the output feature images are transmitted to the pooling layer for feature selection and information filtering, and finally, the classification results of each pixel are output by the output layer to generate the preset model.

[0072] The database / dataset for visual object recognition includes pictures of multiple categories and corresponding labels of each picture, for example, the ImageNet dataset. The structure of ImageNet can include directories, subdirectories, and picture sets in sequence. ImageNet includes more than 20,000 categories, and each category can include hundreds of images.

[0073] It should be understood that the target dataset includes at least one target image, and the subcategory label of the object included in the target image is the same as the subcategory label of the object included in the collection image, that is, the image corresponding to the object information identifiable by the preset model in the collection image can be used as the target image, and the subcategory label corresponding to the object information is also required in the target image.

[0074] In some embodiments, the determination of the target dataset can include the following process: based on the object information, the subcategory label of the object information, and the parent category label corresponding to the subcategory label, an initial dataset is determined, the initial dataset includes at least one initial image; the initial dataset corresponding target dataset is determined by a preset generative adversarial network, and the number of images of the target dataset is greater than the number of images of the initial dataset.

[0075] In some embodiments, the subcategory node corresponding to the object information is determined by the object information, and the subcategory label of the subcategory node is determined; the parent category node corresponding to the subcategory node and the parent category label of the parent category node are determined based on a preset knowledge dataset, wherein the parent category label includes one or more subcategory labels.

[0076] The parent category label can be determined by an existing dataset to determine the parent category node corresponding to the subcategory node, wherein the subcategory node and the parent category node have a mutual relationship; the preset knowledge dataset is used to describe a dataset that has the association characteristics of the subcategory node and the parent category node, for example, the WordNet dataset. WordNet is a dataset that describes the association characteristics of words, and is also a database that associates English nouns, verbs, adjectives, and adverbs with synonyms. These synonyms are related to each other through semantic relationships to determine the definition of a word. The parent category label is generated by searching for the parent category node of the classification subcategory node through the WordNet dataset.

[0077] Figure 2 A schematic diagram of the tree structure of the subcategory node and the parent category node in some embodiments of the present application is shown as follows: Figure 2As shown, the parent class label includes one or more child class labels, the child class label is a specific and detailed label, and the parent class label is a general label corresponding to the child class label, for example, the child class nodes are "teddy cat" and "jade cat", and the corresponding parent class node is "cat". In each edge computing layer, the image of the identified item information and the corresponding child class label and parent class label form a new data set, i.e., an initial data set.

[0078] In some embodiments, a preset generative adversarial network is used to determine similar images corresponding to the initial images with item information; and a target data set is determined based on a target image, the target image including the initial images and the similar images.

[0079] In some embodiments, a preset generative adversarial network is used to determine similar images corresponding to the initial images with item information; and a target data set is determined based on a target image, the target image including the initial images and the similar images.

[0080] In some embodiments, a preset generative adversarial network is used to determine similar images corresponding to the initial images with item information; and a target data set is determined based on a target image, the target image including the initial images and the similar images.

[0081] The parent class model is determined based on the target data set. It should be understood that the target data set can be further determined by a preset model to determine a child class model and a parent class model, and the parent class model is sent to the cloud computing layer.

[0082] In some embodiments, a preset generative adversarial network is used to determine similar images corresponding to the initial images with item information; and a target data set is determined based on a target image, the target image including the initial images and the similar images.

[0083] It should be understood that the pre-training model can be trained based on existing images corresponding to each scene and / or training data composed of databases / datasets for visual object recognition, etc.

[0084] In some embodiments, the training of the pre-training model can train the training data through a convolutional neural network, wherein the images in the training data are normalized at the input layer, the input images are feature extracted at the convolutional layer, the output feature images are transmitted to the pooling layer for feature selection and information filtering, and finally the classification results of each pixel are output by the output layer to generate the pre-training model.

[0085] After the pre-training model identifies the collected image, if there is an unidentifiable second object in the collected image, a continuous learning model is determined based on the image corresponding to the second object, wherein the training of the continuous learning model can expand the input data of the continuous learning model training through a database / dataset for visual object recognition in addition to the collected image corresponding to the second object.

[0086] Through the continuous learning model, an identifiable third object corresponding to the second object is determined, and based on the first object and the third object, the object information is determined, that is, through the training of the continuous learning model, the edge computing layer has the ability to identify new objects.

[0087] As shown in Figure 2 , the cloud computing layer 100 is configured to obtain each parent class model and send each parent class model to the edge computing layer. The cloud computing layer receives the parent class models issued by each edge node of the edge computing layer, converges in the cloud computing layer, and shares each converged parent class model to each edge node. The edge node accepts the parent class model shared from the cloud computing layer, guides each terminal device under the edge node to have the function of identifying new objects, thereby realizing knowledge sharing in different scenarios.

[0088] It should be understood that, based on the process of continuous collaborative learning of the cloud computing layer and the edge computing layer in the cloud-edge architecture, each edge node in the edge computing layer has a continuously updated parent class model, which can improve the identification of new objects by the terminal devices under each edge node and reduce the leakage of user privacy data.

[0089] Figure 3 Another cloud-edge-based architecture is shown in some embodiments of the application, as shown in Figure 3 , the cloud-edge-based architecture includes a terminal device layer 300, an edge computing layer 200 and a cloud computing layer 100, wherein the edge computing layer 200 and the terminal device layer 300 are communicatively connected, and the cloud computing layer 100 and the edge computing layer 200 are communicatively connected.

[0090] As shown in Figure 3As shown, the edge computing layer includes a plurality of edge nodes, such as edge node 210, edge node 220, and edge node 230, each of which includes a training classifier module, a continuous learning module, an edge computing classifier module, and a first communication module, wherein the first communication module is in communication connection with each terminal device of the terminal device layer and in communication connection with the training classifier module, the continuous learning module, and the edge computing classifier module.

[0091] The first communication module in the edge node is configured to obtain the collected images, which can be aggregated from each terminal device of the terminal device layer. The training classifier module is configured to determine a pre-training model based on a known data set. The continuous learning module is configured to determine a continuous learning model based on the image corresponding to the second item and the network image through federated learning. The edge computing classifier module is configured to identify the collected image based on the pre-training model or the continuous learning model to determine the item information in the collected image, and determine an initial data set based on the item information, the sub-class label, and the parent class label of the item information. The initial data set is determined through a preset generative adversarial network, and a target data set corresponding to the initial data set is determined based on the target data set, and a parent class model is determined based on the target data set.

[0092] In some embodiments, the terminal device is a ROS robot, and communication with the first communication module in the edge node can be established through ROS to facilitate the transmission of the collected image to the edge node of the edge computing layer.

[0093] In some embodiments, each terminal device of the terminal device layer is distributed in different geographical locations, and the collected images of different terminal devices can be integrated and aggregated through federated learning (FL) and transmitted to the training classifier module through the first communication module.

[0094] The federated learning is a distributed machine learning technology, the core of which is to perform distributed model training among multiple data sources (i.e., terminal devices) that have local data, to build a global model based on virtual fused data without exchanging local individual or sample data, and to achieve a balance between data privacy protection and data sharing calculation by exchanging only model parameters or intermediate results.

[0095] The determination of the pre-training model in the training classifier module is based on a known data set, which includes a database (data set) for visual object recognition and / or existing images corresponding to each scene. The pre-training model can be trained using the known data set through a convolutional neural network (CNN), and the pre-training model can be merged and aggregated to the edge computing classifier module.

[0096] The continuous learning module can determine a continuous learning model based on the image corresponding to the second item (an item that cannot be recognized by the edge computing classifier module) and the network image through federated learning, and send the continuous learning model to the edge computing classifier module. The continuous learning model is continuously updated to enable the edge computing classifier module in the edge node to have the ability to recognize the second item.

[0097] After the images collected by different terminal devices are integrated and summarized through federated learning, they are transmitted to the edge computing classifier module. The edge computing classifier module first identifies the collected images based on the pre-trained model to determine the item information in the collected images. The first item that can be recognized in the collected images is determined, and it is judged whether the second item that cannot be recognized exists in the collected images. If there is no second item that cannot be recognized in the collected images, the item information is determined based on the first item. If there is a second item that cannot be recognized in the collected images, a continuous learning model is determined based on the image corresponding to the second item. The third item corresponding to the second item is determined through the continuous learning model. And based on the first item and the third item, the item information is determined.

[0098] It should be understood that the item in the collected image can be the first item that can be recognized by the pre-trained model, or the second item that cannot be recognized by the pre-trained model.

[0099] For the second item that cannot be recognized, the third item corresponding to the second item can be recognized through the continuous learning model trained by the continuous learning module, and the item information is determined based on the first item and the third item.

[0100] For the first item that can be recognized by the pre-trained model, and the third item that can be recognized by the continuous learning model but cannot be recognized by the pre-trained model, the item information can be determined. The sub-class label of the item information and the parent class label corresponding to the sub-class label, and the initial data set based on the item information, the sub-class label of the item information and the parent class label corresponding to the sub-class label are determined. The initial data set includes at least one initial image. The initial data set corresponding to the target data set is determined through the GAN generative adversarial network, and the number of images of the target data set is greater than the number of images of the initial data set. Wherein, the similar image corresponding to the initial image with item information is determined through the GAN generative adversarial network. The target data set is determined based on the target image, and the target image includes the initial image and the similar image.

[0101] Wherein, the sub-class node corresponding to the item information and the sub-class label of the sub-class node can be determined through the item information. The parent class node corresponding to the sub-class node and the parent class label of the parent class node are determined based on the WordNet data set.

[0102] In some embodiments, for the first item identifiable by the pre-trained model, and the second item not identifiable by the pre-trained model, but the third item identifiable by the continuous learning model, the item information and the sub-class node corresponding to the item information can be determined, and the parent class node of the item information is found through the WordNet dataset to generate the parent class label, as shown in Figure 2 The tree structure of the sub-class node and the parent class node is established, and the item information and the corresponding sub-class label and parent class label constitute the initial dataset.

[0103] The initial dataset is divided into a training set, a test set, and a validation set. The initial dataset generates a large number of similar images through a GAN (Generative Adversarial Network). The generation network in the GAN randomly samples from the edge node as input, and the output result tries to imitate the real sample (initial image) in the training set. The input of the discriminator network in the GAN can be a real sample or the output of the generation network, thereby generating a large number of picture data similar to the initial image (i.e., similar images). The initial image and the similar image determine the target image, and based on the target image, the target dataset is determined, which realizes the sample expansion of the initial dataset and generates the target dataset.

[0104] The target dataset can further determine the sub-class model and the parent class model through the training model in the training classifier module, and send the parent class model to the cloud computing layer.

[0105] The target dataset generates the sub-class model and the parent class model in the CNN (Convolutional Neural Network) training in the training classifier module. The sub-class model is stored in the edge computing classifier module, and the parent class model is uploaded to the cloud computing layer. The target image has two levels of labels, sub-class labels and parent class labels.

[0106] In some embodiments, there can be one or more terminal devices under each edge node, and the same terminal device can exist under each edge node, or different terminal devices can exist, as shown in Figure 3 The edge node 210 corresponds to the terminal device in the terminal device layer 310, the edge node 220 corresponds to the terminal device in the terminal device layer 320, and the edge node 230 corresponds to the terminal device in the terminal device layer 330.

[0107] As shown in Figure 3 The cloud computing layer and the plurality of edge nodes are in communication connection. The cloud computing layer includes a cloud computing total classifier module, a parent class knowledge continuous learning module, and a second communication module. The second communication module is in communication connection with each edge node of the edge computing layer, and is in communication connection with the cloud computing total classifier module and the parent class knowledge continuous learning module.

[0108] The second communication module in the cloud computing layer is configured to obtain the parent class models of the edge nodes; the parent class knowledge continuous learning model is configured to obtain the continuous learning models of the edge nodes; and the cloud computing total classifier module is configured to update the models in the cloud computing total classifier module according to the parent class models of the edge nodes and the continuous learning models of the edge nodes.

[0109] The parent class models distributed in the edge nodes are converged in the cloud computing total classifier module using federated learning, and the parent class models trained by the edge computing layer are continuously updated to the cloud computing total classifier module through the parent class knowledge continuous learning module.

[0110] The converged parent class models are pushed to the edge nodes that have not obtained new data by the second communication module in the cloud computing layer, and the continuous collaborative learning process of the cloud computing layer and the edge computing layer enables each edge node in the edge computing layer to have continuously updated parent class models, which can improve the identification of new items by the terminal devices under each edge node, thereby realizing knowledge sharing in different scenarios and reducing the leakage of user privacy data.

[0111] Figure 4 The working process of the cloud-edge-end based architecture in some embodiments of the present application is shown in the schematic diagram as shown in Figure 4 As shown, the terminal device in a specific scenario obtains the collected images in the corresponding environment, and the collected images are transmitted to the edge computing classifier module in the edge computing layer to identify the items; it is determined whether the collected item information can be identified, if yes, the process is ended, and if not, the wordnet labeling dataset is used in the edge computing layer. After generating similar data through GAN, the classifier model is trained using federated learning. The edge computing classifier module is updated through the continuous learning module, and the cloud computing total classifier module is updated through the parent class knowledge continuous learning module. It is detected whether the edge computing classifier is updated, if not, the above process is repeated, and if yes, the process is ended.

[0112] The collected images of the terminal device in the terminal device layer in the embodiments of the present application are generated into parent class models through the training classifier module and the edge computing classifier module in the edge computing layer, and are converged into the cloud computing total classifier module in the cloud computing layer. In the process, the terminal device layer to the edge computing layer realizes the conversion from data to model, and the edge computing layer to the cloud computing layer realizes the process from model to model, which reduces the direct uploading of raw data to the cloud computing layer and effectively protects the privacy of users.

[0113] In the embodiments of the present application, the WordNet dataset is used to label the item information, which is different from the traditional dataset labeling method. The detailed item information is labeled as a subclass node, and the abstract concept corresponding to the item information is labeled as a parent class node (for example, Figure 2As shown, only the abstract parent class model is uploaded to the cloud computing layer, and the detailed item information corresponding to the subclass model is saved locally (edge computing layer), thereby effectively protecting the privacy of the user.

[0114] Meanwhile, a large number of similar images are generated by the GAN in the edge computing layer, and the similar data are not original information in the application scenario, but adjusted original information, which can be uploaded to the cloud computing layer without leaking the privacy of the user, thereby protecting the privacy of the user.

[0115] When a new item is collected by the terminal device and the edge computing classifier module fails to identify the item information, the edge computing classifier module is updated through the continuous learning module, and the cloud computing total classifier module is updated through the parent class knowledge continuous learning module of the cloud computing layer, and after the cloud computing total classifier module is updated, the cloud computing layer guides the edge node that does not obtain the new item to update the edge computing classifier module, so as to achieve cloud-edge collaborative continuous learning and realize sharing of knowledge between users, while protecting the privacy of the user.

[0116] Based on the same inventive concept, the embodiment of the present application also provides an item recognition method in an image, Figure 5 A flowchart of an item recognition method in an image in some embodiments of the present application is shown, which is applied to the above-mentioned cloud-edge-terminal based architecture, as shown in Figure 5 As shown, the item recognition method in an image includes the following steps:

[0117] S510, determining a collected image, the collected image including an item to be recognized.

[0118] S520, identifying the collected image through a preset model to determine item information in the collected image.

[0119] S530, determining a target data set based on the item information and a subclass label of the item information, the target data set including at least one target image, and the subclass label of the item included in the target image being the same as the subclass label of the item included in the collected image.

[0120] S540, determining a parent class model based on the target data set.

[0121] Figure 6 A flowchart of determining a target data set in an item recognition method in an image in some embodiments of the present application is shown, as shown in Figure 6 As shown, step 530 of determining a target data set based on the item information and a subclass label of the item information further includes the following steps:

[0122] S531, determining an initial data set based on the item information, the subclass label of the item information and a parent class label corresponding to the subclass label, the initial data set including at least one initial image.

[0123] The sub-class node corresponding to the article information is determined through the article information, and the sub-class label of the sub-class node is determined. The parent class node corresponding to the sub-class node is determined based on a preset knowledge dataset, and the parent class label of the parent class node is determined, wherein the parent class label includes one or more sub-class labels.

[0124] S532, determine the target data set corresponding to the initial data set through the preset generative adversarial network, and the number of images of the target data set is greater than the number of images of the initial data set.

[0125] In a preset model including a pre-training model and a continuous learning model, Figure 7 The flowchart of determining the target data set in the image article recognition method in some embodiments of the present application is shown in FIG. 5. Figure 7 As shown in FIG. 5, the step 520 determines the article information in the collected image by identifying the collected image through the preset model, and further includes the following steps:

[0126] S521, identify the collected image through the pre-training model to determine the first article that can be identified in the collected image.

[0127] S522, determine whether there is a second article that cannot be identified in the collected image.

[0128] S523, if there is no second article that cannot be identified in the collected image, determine the article information based on the first article.

[0129] S524, if there is a second article that cannot be identified in the collected image, determine the continuous learning model through federated learning of the image corresponding to the second article and the network image; determine the third image information corresponding to the second article that cannot be identified through the continuous learning model; and determine the article information based on the first article and the third image information.

[0130] The above-mentioned image article recognition method is applied to the above-mentioned cloud-edge-end based architecture, and has the same inventive concept as the above-mentioned cloud-edge-end based architecture, which will not be described here.

[0131] The following paragraphs will compare and list the Chinese terms involved in the present application and their corresponding English terms for easy reading and understanding.

[0132] For convenience of explanation, the above description has been made in combination with specific embodiments. However, the above description in some embodiments is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are for better explanation of the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A cloud edge-based system, characterized in that, The method comprises the following steps: a terminal device layer configured to acquire a collection image in an environment where each terminal device is located, the collection image comprising an article to be identified; an edge computing layer in communication connection with the terminal device layer, the edge computing layer being configured to aggregate the collection images of each terminal device, identify the collection images through a preset model, determine article information in the collection images, and determine a target data set based on the article information and a sub-class label of the article information, the target data set comprising at least one target image, the sub-class label of an article included in the target image being the same as that of an article included in the collection image, and determine a parent class model based on the target data set; a cloud computing layer in communication connection with the edge computing layer, the cloud computing layer being configured to acquire each parent class model and send each parent class model to the edge computing layer; the edge computing layer is further configured to determine a target data set based on the article information and a sub-class label of the article information, and the edge computing layer is further configured to: determine an initial data set based on the article information, a sub-class label of the article information, and a parent class label corresponding to the sub-class label, the initial data set comprising at least one initial image; determine a target data set corresponding to the initial data set through a preset generative adversarial network, the number of images in the target data set being greater than that in the initial data set; the edge computing layer comprises a plurality of edge nodes, each edge node comprising: a first communication module in communication connection with each terminal device of the terminal device layer, the first communication module being configured to acquire the collection images and aggregate the collection images of each terminal device layer through federated learning; a training classifier module configured to determine a pre-training model based on a known data set; a continuous learning module configured to determine the continuous learning model based on an image corresponding to a second article and a network image through federated learning; an edge computing classifier module in communication connection with the training classifier module, the continuous learning module, and the first communication module, the edge computing classifier module being configured to identify the collection images based on the pre-training model or the continuous learning model, determine article information in the collection images, determine an initial data set based on the article information, a sub-class label of the article information, and a parent class label, and determine a target data set corresponding to the initial data set through a preset generative adversarial network, and determine a parent class model based on the target data set. 2.The cloud edge-based system of claim 1, wherein, the edge computing layer is further configured to determine a similar image corresponding to the initial image with the article information through a preset generative adversarial network; determine the target data set based on the target image, the target image comprising the initial image and the similar image. the edge computing layer is configured to: 3.The cloud edge-based system of claim 1, wherein, ​ Determine a sub-class node corresponding to the item information and a sub-class label of the sub-class node through the item information; Determine a parent class node corresponding to the sub-class node and a parent class label of the parent class node based on a preset knowledge dataset, wherein the parent class label includes one or more sub-class labels.

4. The cloud edge-based system of claim 1, wherein, The preset model includes a pre-training model and a continuous learning model. The edge computing layer is further configured to: Determine a first item that can be recognized in the collected image through the pre-training model, and determine whether a second item that cannot be recognized exists in the collected image; If the collected image does not include the second item that cannot be recognized, determine the item information based on the first item; If the collected image includes the second item that cannot be recognized, determine the continuous learning model based on an image corresponding to the second item, determine a third item that can be recognized corresponding to the second item through the continuous learning model, and determine the item information based on the first item and the third item.

5. The cloud edge-based system of claim 1, wherein, The cloud computing layer and the plurality of edge nodes are in communication connection. The cloud computing layer includes: A second communication module in communication connection with each edge node of the edge computing layer, configured to acquire the parent class model of each edge node; A parent class knowledge continuous learning module in communication connection with the second communication module, configured to acquire the continuous learning model of each edge node; A cloud computing total classifier module in communication connection with the parent class knowledge continuous learning module and the second communication module, configured to update the model in the cloud computing total classifier module according to the parent class model of each edge node and the continuous learning model of each edge node.

6. An article recognition method in an image, characterized by, The cloud edge-based system of claim 1 includes: Acquire a collected image, the collected image including an item to be recognized; Determine item information in the collected image through a preset model; Determine a target dataset based on the item information and a sub-class label of the item information, the target dataset including at least one target image, a sub-class label of an item included in the target image being the same as a sub-class label of an item included in the collected image; Determine a parent class model based on the target dataset; The determination of the target dataset based on the item information and the sub-class label of the item information includes: Determine an initial dataset based on the item information, a sub-class label of the item information, and a parent class label corresponding to the sub-class label, the initial dataset including at least one initial image; Determine a target dataset corresponding to the initial dataset through a preset generative adversarial network, the number of images in the target dataset being greater than the number of images in the initial dataset.

7. The method of claim 6, wherein, The preset model includes a pre-training model and a continuous learning model, and the collected image is identified through the preset model to determine the item information in the collected image, including: The collected image is identified through the pre-training model to determine a first item that can be identified in the collected image, and it is determined whether a second item that cannot be identified exists in the collected image; If the collected image does not contain the second item that cannot be identified, the item information is determined based on the first item; If the collected image contains the second item that cannot be identified, the image corresponding to the second item and the network image are determined through federated learning to determine the continuous learning model; the third image information corresponding to the second item that cannot be identified is determined through the continuous learning model; and the item information is determined based on the first item and the third image information.

Images