Training method of neural network model for image processing, image recognition method and device, electronic equipment and medium

Abstract

The invention provides a training method of a neural network model for image processing, an image recognition method and device, electronic equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical field of image recognition. The neural network model comprises a first sub-model and a second sub-model, and the training method of the first sub-model comprises the following steps: acquiring a first sample image, and marking a real coordinate value of a region of interest in the first sample image; inputting the first sample image into the first sub-model to obtain a first output; adjusting parameters of the first sub-model; the training method of the second sub-model comprises the following steps: acquiring a second sample image, and marking a real threshold value; inputting the second sample image into the first sub-model to obtain a second output of the first sub-model; inputting the second output into the second sub-model, and obtaining a prediction threshold value output by the second sub-model; and adjusting parameters of the second sub-model based on the real threshold value and the prediction threshold value.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of image recognition technology, and specifically to a training method for a neural network model for image processing, an image recognition method, device, electronic device, computer-readable cache medium, and computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] In the fields of catering stores, warehousing, logistics, etc., densely stacked items such as tableware and goods need to be counted quickly and accurately. This kind of counting usually relies on manual operation, which is not only time-consuming and labor-intensive, but also easily affected by human factors, resulting in inaccurate counting results. With the development of image processing technology, image-based automatic counting methods have gradually become a research hotspot. How to process images more effectively so that fast and accurate counting can be achieved based on the processed images has become a problem to be solved.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a training method for a neural network model for image processing, a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for image recognition.

[0006] According to one aspect of the present disclosure, a training method for a neural network model for image processing is provided, wherein the neural network model includes a first sub-model and a second sub-model, and wherein the training method includes a training method for the first sub-model and a training method for the second sub-model, the training method for the first sub-model including: obtaining a first sample image containing a first object to be counted, and marking the real coordinate value of the region of interest in the first sample image; inputting the first sample image into the first sub-model, obtaining a first output of the first sub-model, the first output representing the predicted coordinate value of the region of interest in the first sample image; adjusting the parameters of the first sub-model based on the real coordinate value and the predicted coordinate value, wherein the training method for the second sub-model includes: in response to completion of the training of the first sub-model, obtaining a first sample image containing a second object to be counted, and marking the real coordinate value of the region of interest in the first sample image; inputting the first sample image into the first sub-model, and obtaining a first output of the first sub-model, the first output representing the predicted coordinate value of the region of interest in the first sample image; adjusting the parameters of the first sub-model based on the real coordinate value and the predicted coordinate value, wherein the training method for the second sub-model includes: in response to completion of the training of the first sub-model, obtaining a first sample image containing a second object to be counted, and marking the real coordinate value of the region of interest in the first sample image; A second sample image of the product is inputted into the first sub-model, and a true threshold for binarizing the second sample image is marked, wherein the outline of the second object to be counted in the second sample image after binarization based on the true threshold meets the clarity standard; the second sample image is inputted into the first sub-model to obtain a second output of the first sub-model, wherein the second output represents the predicted coordinate value of the region of interest of the second sample image; the second output is inputted into the second sub-model to obtain the predicted threshold for binarizing the second sample image outputted by the second sub-model; a loss value is calculated based on the true threshold and the predicted threshold; and parameters of the second sub-model are adjusted based on the loss value, wherein the second sample image can be used to identify and count the second object to be counted contained therein after being binarized based on the predicted threshold.

[0007] According to another aspect of the present disclosure, there is provided a method for image recognition, comprising: acquiring a first image containing objects to be counted, wherein in the first image, the objects to be counted are stacked along a first direction; determining a threshold for binarizing the first image using a neural network model; binarizing the first image based on the threshold to obtain a second image; setting a first sliding window in the second image, and sliding the first sliding window M times along a second direction in the second image, wherein the length of the first sliding window in the first direction is not less than the length of the second image in the first direction, and wherein the second direction is perpendicular to the first direction, and wherein M is an integer greater than 1; identifying the number Cnti of the objects to be counted contained in the first sliding window in each sliding, wherein i∈[1,M]; and determining the number of the objects to be counted in the first image based on the number Cnti corresponding to each sliding of the first sliding window, wherein the neural network model is trained according to the aforementioned training method for the neural network model for image processing.

[0008] According to another aspect of the present disclosure, there is provided a training device for a neural network model for image processing, wherein the neural network model includes a first sub-model and a second sub-model, and wherein the training device includes a training device for the first sub-model and a training device for the second sub-model, the training device for the first sub-model includes: a first acquisition module, configured to acquire a first sample image containing a first object to be counted, and annotate the real coordinate value of the region of interest in the first sample image; a second acquisition module, configured to input the first sample image into the first sub-model, and acquire a first output of the first sub-model, wherein the first output represents the predicted coordinate value of the region of interest in the first sample image; a first adjustment module, configured to adjust the parameters of the first sub-model based on the real coordinate value and the predicted coordinate value, wherein the training device for the second sub-model includes: a third acquisition module, configured to acquire a first sample image containing a first object to be counted in response to completion of the training of the first sub-model, and annotate the real coordinate value of the region of interest in the first sample image; a second sample image of an object, and annotating a true threshold for binarizing the second sample image, wherein the contour of the second object to be counted in the second sample image after binarization based on the true threshold satisfies a clarity standard; a fourth acquisition module, configured to input the second sample image into the first sub-model, and obtain a second output of the first sub-model, wherein the second output represents the predicted coordinate value of the region of interest of the second sample image; a sixth acquisition module, configured to input the second output into the second sub-model, and obtain the predicted threshold output by the second sub-model for binarizing the second sample image; a calculation module, configured to calculate a loss value based on the true threshold and the predicted threshold; and a second adjustment module, configured to adjust the parameters of the second sub-model based on the loss value, wherein the second sample image can be used to identify and count the second object to be counted contained therein after being binarized based on the predicted threshold.

[0009] According to another aspect of the present disclosure, there is provided an image recognition device, comprising: a seventh acquisition module, configured to acquire a first image containing objects to be counted, wherein in the first image, the objects to be counted are stacked along a first direction; a second determination module, configured to determine a threshold for binarizing the first image using a neural network model; an image processing module, configured to binarize the first image based on the threshold to obtain a second image; a first sliding module, configured to set a first sliding window in the second image, and slide the first sliding window M times along a second direction in the second image, wherein the first sliding window The length of the window in the first direction is not less than the length of the second image in the first direction, and wherein the second direction is perpendicular to the first direction, and wherein M is an integer greater than 1; a third determination module is configured to identify the number Cnti of the objects to be counted contained in the first sliding window in each sliding, wherein i∈[1,M]; and a fourth determination module is configured to determine the number of the objects to be counted in the first image based on the number Cnti corresponding to each sliding of the first sliding window, wherein the neural network model is trained according to the aforementioned training method of the neural network model for image processing.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any of the above methods.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any of the above methods.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements any of the above methods when executed by a processor.

[0013] According to one or more embodiments of the present disclosure, a method for training a neural network model for image processing is provided, wherein the trained neural network model is used to predict a region of interest in an image and can predict a threshold for binarizing the image. The neural network model is used to determine the region of interest and the threshold for binarizing the image, and the region of interest and the threshold can be dynamically adjusted and determined based on the actual image. The image is binarized based on the threshold obtained by dynamic prediction so that the outlines of objects to be counted in the processed image are clearer, thereby improving the accuracy of counting the objects in the processed image subsequently.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to an exemplary embodiment.

[0017] Figure 2 A flowchart of a training method for a neural network model for image processing according to an embodiment of the present disclosure is shown;

[0018] Figure 3 shows an architectural diagram of a first sub-model according to an embodiment of the present disclosure;

[0019] Figure 4 shows an architectural diagram of a second sub-model according to an embodiment of the present disclosure;

[0020] Figure 5 A flowchart of an image recognition method according to an embodiment of the present disclosure is shown;

[0021] Figure 6 A schematic diagram of performing image processing using a neural network model according to an embodiment of the present disclosure is shown;

[0022] Figure 7 A structural block diagram of a training device for a neural network model for image processing according to an embodiment of the present disclosure is shown;

[0023] Figure 8A structural block diagram of an image recognition device according to an embodiment of the present disclosure is shown; and

[0024] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0027] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0028] In the related art, a detection / segmentation model can be used to detect / segment the objects to be counted, and then the number of objects is obtained by counting the results of the detection / segmentation. However, this counting scheme is suitable for counting scenes with regular cross-sectional shapes and clear and regular boundaries of objects, such as the counting of objects such as bamboo sticks and steel bars. In scenes where the cross-sections of objects are irregular and densely packed, the counting accuracy of the above counting scheme will be seriously reduced because the boundaries of the objects cannot be accurately determined. In some related technologies, the counting of objects can be performed based on the scheme of traditional image processing methods, specifically, the image is first denoised, the image is binarized, and then the objects to be counted are found through morphological corrosion and expansion, and operations such as finding connected domains are performed, and finally the number of objects is counted. However, this scheme is suitable for scenes with relatively simple backgrounds, and the threshold used for the binarized image is a fixed value. However, the actual scenes are complex and changeable, and the use of a fixed threshold for binarization will not be able to adapt to the changing scenes. Therefore, the fixed threshold scheme lacks flexibility and has poor robustness.

[0029] To solve the above problems, the present disclosure provides a training method for a neural network model for image processing. The trained neural network model is used to predict the region of interest in the image and can predict the threshold for binarizing the image. The neural network model is used to determine the region of interest and the threshold for binarizing the image. The region of interest and the threshold can be dynamically adjusted and determined according to the actual image. The image is binarized based on the threshold obtained by dynamic prediction so that the outline of the objects to be counted in the processed image is clearer, thereby improving the counting accuracy when the objects in the processed image are subsequently counted.

[0030] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0032] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of a training method for a neural network model for image processing or a method for image recognition.

[0033] In some embodiments, server 120 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0034] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1is an example of a system for implementing the methods described herein and is not intended to be limiting.

[0035] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to perform a training method for a neural network model for image processing or a method for image recognition. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0036] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0037] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0038] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0039] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0040] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0041] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0042] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0043] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0044] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0045] Figure 2 A flowchart of a method for training a neural network model for image processing according to an embodiment of the present disclosure is shown, wherein the neural network model includes a first sub-model and a second sub-model.

[0046] The training method 200 of the neural network model for image processing includes the training method of the first sub-model and the training method of the second sub-model, such as Figure 2 As shown, the training method of the first sub-model includes:

[0047] Step S201, obtaining a first sample image containing a first object to be counted, and marking the real coordinate values ​​of the region of interest in the first sample image;

[0048] Step S202: input the first sample image into the first sub-model to obtain a first output of the first sub-model, where the first output represents predicted coordinate values ​​of a region of interest in the first sample image;

[0049] Step S203: adjusting the parameters of the first sub-model based on the real coordinate value and the predicted coordinate value;

[0050] The training method of the second sub-model includes:

[0051] Step S204: in response to the completion of the training of the first sub-model, obtaining a second sample image containing second objects to be counted, and marking a true threshold for binarizing the second sample image, wherein the contour of the second objects to be counted in the second sample image after binarization based on the true threshold meets a clarity standard;

[0052] Step S205: input the second sample image into the first sub-model to obtain a second output of the first sub-model, wherein the second output represents predicted coordinate values ​​of the region of interest of the second sample image;

[0053] Step S206: input the second output into the second sub-model to obtain a prediction threshold output by the second sub-model for binarizing the second sample image;

[0054] Step S207: Calculate a loss value based on the real threshold and the predicted threshold; and

[0055] Step S208: adjusting the parameters of the second sub-model based on the loss value, wherein the second sample image can be used to identify and count the second objects to be counted contained therein after being binarized based on the prediction threshold.

[0056] The training method 200 of the neural network model for image processing includes two stages. First, in the first stage, the first sub-model is trained to predict the region of interest in the image. It can be understood that the first object to be counted may be located in a partial area of ​​the first sample image. Therefore, predicting the region of interest in the image containing the first object to be counted can focus the subsequent calculation process on the area where the object may exist, rather than processing the entire image, which helps to reduce the amount of calculation, thereby improving the efficiency of subsequent object counting based on the processed image. In addition, the prediction of the region of interest helps to exclude background and interference factors in the image that are not related to the object to be counted, thereby reducing the possibility of misidentification and miscounting.

[0057] The first sub-model in the neural network model is obtained by training to predict the region of interest in the image. Multiple sample images in various image scenes can be used to train the first sub-model, which can enable it to have better generalization ability and adapt to different backgrounds and object arrangements, so as to achieve accurate recognition of the region of interest in various practical applications, and provide a reliable foundation for subsequent image processing steps (such as binarization, object counting based on processed images, etc.), ensuring that these steps can be performed in the appropriate area, thereby improving the overall performance of the entire object counting.

[0058] In the second stage of model training, the second sub-model is trained in combination with the predicted coordinate values ​​of the region of interest output by the first sub-model, so that the second sub-model can focus on processing the region of interest without having to search in the entire image, which significantly reduces computational complexity and resource consumption and improves training efficiency. In addition, the second sub-model can learn and predict the optimal threshold for binarization by training on a large number of labeled sample images. This method is based on the true threshold in the training data, so that the model can automatically adjust the threshold to adapt to the contour, brightness and contrast changes of different types of objects to be counted in different images, so that it can still generate clear and accurate binarized images when facing different images and object arrangements, thereby providing high-quality input for subsequent object recognition and counting.

[0059] Thus, a neural network model including a first sub-model and a second sub-model is obtained by training the training method 200 of the neural network model for image processing, which are used to predict the region of interest in the image and the threshold for binarizing the image, respectively. The neural network model can dynamically adjust and accurately determine the region of interest according to the actual image. The neural network model makes the boundaries of objects in the binary image clearer by dynamically adjusting the binarization threshold. Especially in complex scenes, such as stacked objects, large changes in lighting or complex backgrounds, the model can automatically generate the optimal threshold for each specific image. This dynamic adjustment avoids the errors that may be caused by the fixed threshold method, thereby ensuring that the binary image can accurately reflect the boundary between the object and the background, making the binarization result of the image clearer and more accurate.

[0060] In specific applications, the prediction thresholds generated by the model can significantly improve the quality of the binary image, which directly affects the accuracy of object counting. The clear object outlines enable subsequent recognition and counting methods to more accurately identify the boundaries of the objects, avoiding miscounting due to blurred or incomplete boundaries. In this way, the model ensures that the outlines of objects can be clearly segmented in various scenarios, thereby improving the overall accuracy of counting.

[0061] In addition, during the training process, the second sub-model can learn the binarization thresholds of various sample images to provide stable output under different types of images and object arrangements. This adaptive binarization processing method significantly improves the generalization ability of the model, enabling it to output high-quality binarization results when facing different image conditions, ensuring the accuracy of counting.

[0062] According to some embodiments, the region of interest is the smallest rectangular region in the image that can contain the objects to be counted in the image, thereby reducing the interference of redundant background in the image and enhancing the accuracy of object counting based on the image.

[0063] According to some embodiments, the first sub-model comprises a feature extraction network for extracting a feature map of an input image.

[0064] According to some embodiments, step S206 includes: obtaining a feature map extracted by the feature extraction network of the first sub-model for the second sample image; inputting the second output and the feature map into the second sub-model, and obtaining the prediction threshold output by the second sub-model for binarizing the second sample image.

[0065] Therefore, the feature map is extracted by the feature extraction network of the first sub-model, and the second sub-model is trained in combination with the predicted coordinate values ​​of the region of interest output by the first sub-model.

[0066] According to some embodiments, the feature map extracted by the feature extraction network of the first sub-model for the input image includes multiple feature maps with different scales.

[0067] Figure 3 FIG. 4 shows an architecture diagram of a first sub-model according to an embodiment of the present disclosure. Figure 3 As shown, the first sub-model 300 includes: a reference network 301 composed of multiple convolution modules, receiving an input image; a feature extraction network 302, used to extract feature maps of different scales from the input image, wherein the feature maps of different scales are spliced ​​along the channel; and a detection head 303, used to output the output of the model.

[0068] Exemplarily, feature maps of different scales may include a feature map P3 of 1 / 8 the size of the original image, a feature map P4 of 1 / 16 the size of the original image, a feature map P5 of 1 / 32 the size of the original image, and a feature map P6 of 1 / 64 the size of the original image.

[0069] Therefore, the use of feature maps of different scales can capture different levels of information in the image. Feature maps of larger scales (such as P3) retain more spatial detail information and are suitable for detecting larger objects or structures in a larger range; while feature maps of smaller scales (such as P6) focus on more abstract, global features, which help to identify the overall outline and background of objects. This combination of multi-scale feature maps enables the model to consider both local details and global context at the same time, improving the model's perception ability. In addition, by combining multi-scale feature maps, the model can better cope with objects of various scales and different image resolutions. Regardless of whether the object is large or small, near or far in the image, the combination of multi-scale feature maps can ensure the stability of the model in different scenarios and reduce recognition and counting errors caused by changes in object size. At the same time, it enables the model to effectively cope with complex scenes and improve the generalization ability of processing diverse images.

[0070] In some examples, the above network can be compressed, for example, by reducing the number of channels of feature maps at each stage in the network, reducing the number of basic module stacks at each stage of the network, etc., to improve the inference speed of the first sub-model.

[0071] According to some embodiments, during the training of the second sub-model, the parameters of the first sub-model remain fixed. During the training of the second sub-model, the reference network 301 and the feature extraction network 302 of the first sub-model can be reused, and the parameters therein are frozen and not updated, so only the parameters of the second sub-model are updated during training. Thus, the second sub-model can be more focused on the accuracy of threshold prediction and improve the effect of binarization processing.

[0072] Based on the training method of the first sub-model, the training method 200 of the neural network model for image processing also includes a training method of the second sub-model for predicting a threshold value for binarizing the image.

[0073] Figure 4 FIG. 4 shows an architecture diagram of a second sub-model according to an embodiment of the present disclosure. Figure 4 As shown, the second sub-model 400 includes a scaling layer 401, a feature fusion layer 402, a pooling layer 403 and a fully connected layer 404. Step S207 includes: inputting a plurality of feature maps (P3, P4, P5, P6) with different scales into the scaling layer 401, and obtaining a feature map with the same scale output by the scaling layer 401; inputting the feature map with the same scale into the feature fusion layer 402 to perform channel splicing on the feature map with the same scale, and obtaining a fused feature map output by the feature fusion layer 402; inputting the fused feature map and the second output into the pooling layer 403 to map the predicted coordinate value of the minimum rectangular area that can contain the items to be counted in the second sample image to the fused feature map, and obtaining a pooled feature map output by the pooling layer 403; inputting the pooled feature map into the fully connected layer 404, and obtaining the prediction threshold output by the fully connected layer 404.

[0074] The input of the second sub-model is the feature maps (P3, P4, P5, P6) of each scale in the feature extraction network 302 in the second sub-model. Exemplarily, the scaling layer 401 of the second sub-model first scales each feature map to 1 / 16 of the original image size, that is, the scale size of the feature map P4. Then, the feature fusion layer 502 splices the features along the channel dimension. The feature maps of the same scale ensure that the features are aligned at the pixel positions corresponding to each channel, so as to ensure that the spliced ​​feature maps maintain spatial consistency and facilitate subsequent convolution or pooling operations. By adjusting the scale of the feature map to make it consistent, it can be ensured that the spatial information in the feature map retains its integrity and accuracy after splicing. The features are fused using a convolution with a convolution kernel size of 1x1, and the channel is scaled to 412 to obtain a fused feature map. This process can be expressed by the following formula:

[0075] F i =f s (P i ),3≤i≤6

[0076] F=Conv 1x1 (F3||F4||F5||F6)

[0077] Among them, f s (·) represents scaling, Conv 1x1 (·) represents the convolution operation, and || represents concatenation along the channel.

[0078] Exemplarily, the second output of the first sub-model represents the predicted coordinate value of the region of interest of the second sample image, which can be recorded as B(x1, y1, x2, y2). The pooling layer 403 can map the predicted coordinate value of the region of interest of the second sample image to the fused feature map, and pool the region of interest into a size of S×S to output the pooled feature map F b . b After flattening, it is sent to the fully connected layer 404, which finally outputs the predicted threshold. This process can be expressed by the following formula:

[0079] F b =G b (F)

[0080] P out =G f (F b )

[0081] Among them, G b (·) represents the operation of feature clipping and pooling, G f (·) is a fully connected layer.

[0082] like Figure 4The hierarchical structure design shown (scaling layer, feature fusion layer, pooling layer and fully connected layer) can more effectively utilize the multi-scale features of the image. By inputting feature maps of different scales (P3, P4, P5, P6) into the scaling layer 401 and scaling them to the same scale, the size differences between feature maps of different scales can be eliminated, making the subsequent feature fusion operation more unified. This operation ensures that the features learned by the model on feature maps of different scales can be effectively compared and combined at the same scale, thereby improving the accuracy of threshold prediction. The predicted coordinate values ​​of the region of interest are mapped to the fused feature map through the pooling layer 403, and the region of interest is pooled into a feature map of a fixed size, thereby achieving accurate positioning of the items to be counted.

[0083] Therefore, the combination of scaling, feature fusion, pooling and fully connected layers enables the second sub-model to process not only image features of a single scale, but also feature maps from multiple scales. This multi-scale fusion mechanism enables the model to maintain good generalization capabilities when facing various image resolutions and object sizes, improves the accuracy of threshold prediction, and ensures the robustness of binarization processing in complex image scenes. In turn, it can improve the accuracy of subsequent object counting based on the processed images.

[0084] According to another aspect of the present disclosure, a method for image recognition is provided. Figure 5 As shown, the image recognition method 500 includes:

[0085] Step S501: acquiring a first image containing objects to be counted, wherein in the first image, the objects to be counted are stacked along a first direction;

[0086] Step S502: using a neural network model to determine a threshold for binarizing the first image;

[0087] Step S503: binarize the first image based on the threshold value to obtain a second image;

[0088] Step S504: setting a first sliding window in the second image, and making the first sliding window slide M times along a second direction in the second image, wherein a length of the first sliding window in the first direction is not less than a length of the second image in the first direction, and wherein the second direction is perpendicular to the first direction, and wherein M is an integer greater than 1;

[0089] Step S505: Identify the number Cnt of the objects to be counted contained in the first sliding window in each sliding operation. i , where i∈[1,M]; and

[0090] Step S506: based on the number Cnt corresponding to each sliding of the first sliding window i , determining the number of the objects to be counted in the first image, wherein the neural network model is trained according to a training method for a neural network model for image processing.

[0091] According to some embodiments, step S501 includes: photographing the objects to be counted to obtain an original image; determining a region corresponding to the objects to be counted in the original image, and capturing the region as the first image.

[0092] Therefore, by determining the specific area where the objects to be counted are located and capturing it as the second image, the interference of irrelevant background is reduced and the accuracy of subsequent image processing is improved.

[0093] Exemplarily, in the first image, the objects to be counted are stacked along a first direction. Exemplarily, the first direction is a vertical direction, and the second direction is a horizontal direction.

[0094] Step S502 uses a neural network model to determine a threshold for binarizing the first image, and can adaptively determine the binarization threshold to obtain an optimal binarization effect, thereby reducing the error of manually setting the threshold. Thus, the flexibility and adaptability of the image binarization process are guaranteed, so that the binarization result can better adapt to different image scenes, and improve the accuracy of subsequent object counting.

[0095] Step S503 separates the objects to be counted from the background in the image through binarization processing, so that the objects in the image are more prominent, which improves the effectiveness of image analysis and facilitates subsequent object counting.

[0096] Steps S504 to S506 set a sliding window and slide it in the image. The sliding window slides and counts in a direction perpendicular to the direction in which the objects are placed. The number of objects can be counted at different positions to prevent the objects from sticking together at some positions and affecting the accuracy of the counting, thus effectively reducing local counting errors. The comprehensiveness and accuracy of the counting are ensured by accumulating statistics through multiple sliding movements.

[0097] According to some embodiments, the region is the smallest rectangular region in the original image that can contain the objects to be counted, thereby reducing interference from redundant background in the image and enhancing the accuracy of object counting based on the image.

[0098] According to some embodiments, determining the area corresponding to the object to be counted in the original image includes: inputting the original image into the neural network model, and obtaining the coordinate value of the area output by the neural network model. Thus, by directly obtaining the coordinate value of the area of ​​interest in the image using the neural network model, the accuracy of area positioning can be improved, the error of manual marking can be avoided, and the degree of automation of the counting scheme can be further improved.

[0099] Therefore, by using a neural network model to determine the region of interest and the threshold for binarizing the image, the region of interest and the threshold can be dynamically adjusted and determined according to the actual image. Binarizing the image based on the threshold obtained by dynamic prediction can improve the accuracy of object counting when subsequently counting objects in the above-mentioned processed image.

[0100] According to some embodiments, before binarizing the first image, the method 500 for image recognition further includes: converting the first image into a grayscale image, and performing median filtering on the grayscale image. Converting the first image into a grayscale image can remove color information, simplify image content, and thus reduce computational complexity. In addition, the grayscale image retains necessary brightness information, which is helpful for subsequent binarization processing. Performing median filtering on the grayscale image can effectively remove noise and isolated points in the image, further improving the smoothness of the image. This processing reduces the impact of noise on the binarization result, ensures more accurate threshold determination, and helps improve the accuracy of object counting.

[0101] According to some embodiments, before setting the sliding window in the second image, the method 500 for image recognition further includes: median filtering the second image and performing a morphological erosion operation. After binarization, median filtering the second image helps to further smooth the binary image and reduce small noise points that may be introduced in the binarization process, thereby ensuring the continuity and integrity of the contour. The morphological erosion operation is used to remove small interferences and loosely connected pixels in the image, further optimizing the contour shape of the object, making it more accurate to identify the number of objects in the sliding window. The erosion operation can also effectively reduce the phenomenon of misconnection between objects, ensure the independence of each object, and improve the accuracy of counting.

[0102] These preprocessing steps effectively improve the stability and robustness of the image recognition method, ensuring that accurate object counting results can be obtained in complex and noisy scenes. Through noise reduction, smoothing and morphological processing, the impact of noise and bad pixels on counting is reduced, which greatly improves the accuracy of subsequent sliding window statistics, making the entire image recognition method more robust and reliable.

[0103] Figure 6FIG. 2 shows a schematic diagram of image processing using a neural network model according to an embodiment of the present disclosure. Figure 6 As shown, the neural network model includes a first sub-model 601 and a second sub-model 602. The first sub-model 601 includes a reference network 603, a feature extraction network 604, and a detection head 605, which is used to output the coordinate value of the minimum rectangular area containing the items to be counted. The second sub-model 602 reuses the feature extraction network of the first sub-model, and predicts and outputs the threshold for binarization based on the coordinate value output by the first sub-model 601.

[0104] According to some embodiments, step S504 includes: sliding the first sliding window M times along the second direction in the second image with a first step length, so as to slide from a first edge of the second image to a second edge of the second image.

[0105] A first sliding window is set in the second image and is slid along the second direction from the first edge of the second image to the second edge of the second image, ensuring that all features of the objects to be counted are captured at different positions. By ensuring that the length of the sliding window in the first direction is not less than the length of the image, the entire image can be covered in the first direction, thereby ensuring that no objects to be counted are missed. This setting helps to improve the comprehensiveness and accuracy of counting and avoid errors caused by uneven distribution of objects in the image.

[0106] According to some embodiments, the image recognition method 500 also includes: setting a second sliding window in the second image, wherein the length of the second sliding window in the first direction is not less than the length of the first sliding window in the first direction; before causing the first sliding window to start sliding, causing the second sliding window to slide N times in the first sliding window along the second direction with a second step length, wherein N is an integer greater than 1; and after each of the M slidings of the first sliding window, causing the second sliding window to slide N times in the first sliding window along the second direction with the second step length.

[0107] According to some embodiments, causing the second sliding window to slide N times in the first sliding window along the second direction with a second step length includes: causing the second sliding window to slide N times in the first sliding window along the second direction with the second step length to slide from a first edge of the first sliding window to a second edge of the first sliding window.

[0108] Therefore, by setting a second sliding window within the first sliding window for refined sliding, the precision of counting is further improved. The operation of the second sliding window helps to capture smaller local changes and details, avoiding missing smaller objects or counting errors in complex scenes. The combined use of this multi-layer sliding window can significantly improve the accuracy and robustness of object counting in the image.

[0109] According to some embodiments, step S505 includes: determining the number Cnt of the objects to be counted contained in the second sliding window in each sliding of the second sliding window during the process in which the second sliding window slides from the first edge of the first sliding window to the second edge of the first sliding window. j , where j∈[1,N]; and the number Cnt corresponding to each sliding of the second sliding window j , determine the number Cnt of the objects to be counted contained in the first sliding window i .

[0110] According to some embodiments, the number Cnt corresponding to each sliding of the second sliding window is j , determine the number Cnt of the objects to be counted contained in the first sliding window i The process includes: in the process where the second sliding window slides from the first edge of the first sliding window to the second edge of the first sliding window, based on the number Cnt corresponding to each sliding of the second sliding window j The mode of the first sliding window is used to determine the number of the objects to be counted contained in the first sliding window. i .

[0111] During the sliding process of the second sliding window, the number of local objects is determined based on the mode of each sliding, which can effectively reduce the false counting caused by noise or irregular shapes in the image. As a statistical method, the mode can provide a stable counting method to avoid the influence of extreme values ​​on the results, thereby improving the stability and reliability of counting.

[0112] According to some embodiments, step S506 includes: in the process of sliding the first sliding window from the first edge of the second image to the second edge of the second image, based on the number Cnt corresponding to each sliding of the first sliding window i The maximum value among them is used to determine the number of the objects to be counted in the first image.

[0113] During the sliding process of the first sliding window, the number of objects in the overall image is determined based on the maximum value of each sliding, which can ensure that the final result fully takes into account the objects in possible overlapping or dense areas, thereby minimizing omissions and improving the accuracy of the counting results.

[0114] Therefore, the image recognition method 500 further refines the counting process by introducing the sliding window counting method, making the counting result more accurate, while reducing possible omissions in the counting process, thereby improving the stability and accuracy of object counting.

[0115] According to another aspect of the present disclosure, a training device for a neural network model for image processing is provided, wherein the neural network model includes a first sub-model and a second sub-model, the first sub-model is used to predict a region of interest in an image, and the second sub-model is used to predict a threshold for binarizing the image.

[0116] like Figure 7 As shown, the training device 700 of the neural network model for image processing includes a training device 701 for the first sub-model and a training device 702 for the second sub-model.

[0117] The training device 701 of the first sub-model includes: a first acquisition module 701-1, configured to acquire a first sample image containing a first object to be counted, and mark the real coordinate values ​​of the area of ​​interest in the first sample image; a second acquisition module 701-2, configured to input the first sample image into the first sub-model, and acquire a first output of the first sub-model, wherein the first output represents the predicted coordinate values ​​of the area of ​​interest in the first sample image; and a first adjustment module 701-3, configured to adjust the parameters of the first sub-model based on the real coordinate values ​​and the predicted coordinate values.

[0118] The training device 702 of the second sub-model includes: a third acquisition module 702-1, configured to acquire a second sample image containing a second object to be counted in response to the completion of the training of the first sub-model, and mark a true threshold for binarizing the second sample image, wherein the contour of the second object to be counted in the second sample image after binarization based on the true threshold satisfies the clarity standard; a fourth acquisition module 702-2, configured to input the second sample image into the first sub-model, and acquire a second output of the first sub-model, wherein the second output represents the predicted coordinate value of the region of interest of the second sample image; a sixth acquisition module 702-4, configured to input the second output into the second sub-model, and acquire the predicted threshold output by the second sub-model for binarizing the second sample image; a calculation module 702-5, configured to calculate a loss value based on the true threshold and the predicted threshold; and a second adjustment module 702-6, configured to adjust the parameters of the second sub-model based on the loss value, wherein the second sample image can be used to identify and count the second object to be counted contained therein after binarization based on the predicted threshold.

[0119] The training device 700 of the neural network model for image processing includes a training device 701 of a first sub-model and a training device 702 of a second sub-model. It is understandable that the first object to be counted may be located in a partial area of ​​the first sample image. Therefore, the training device 701 of the first sub-model trains the first sub-model to predict the region of interest containing the first object to be counted in the image, so that the subsequent calculation process can be concentrated in the area where the object may exist, rather than processing the entire image, which helps to reduce the amount of calculation, thereby improving the efficiency of subsequent object counting based on the processed image. In addition, the prediction of the region of interest helps to exclude background and interference factors in the image that are not related to the object to be counted, thereby reducing the possibility of misidentification and miscounting.

[0120] The training device 701 of the first sub-model obtains the first sub-model in the neural network model through training to predict the region of interest in the image. Multiple sample images in multiple image scenes can be used to train the first sub-model, which can enable it to have better generalization ability and adapt to different backgrounds and object arrangements, so as to achieve accurate identification of the region of interest in various practical applications, and provide a reliable foundation for subsequent image processing steps (such as binarization, object counting based on processed images, etc.), ensuring that these steps can be performed in the appropriate area, thereby improving the overall performance of the entire object counting.

[0121] The training device 702 of the second sub-model trains the second sub-model in combination with the predicted coordinate values ​​of the region of interest output by the first sub-model, so that the second sub-model can focus on processing the region of interest without having to search in the entire image, which significantly reduces the computational complexity and resource consumption and improves the training efficiency. In addition, the second sub-model can learn and predict the optimal threshold for binarization by training a large number of labeled sample images. This method is based on the real threshold in the training data, so that the model can automatically adjust the threshold to adapt to the contour, brightness and contrast changes of different types of objects to be counted in different images, so that it can still generate clear and accurate binarized images when facing different images and object arrangements, thereby providing high-quality input for subsequent object recognition and counting.

[0122] Thus, a neural network model including a first sub-model and a second sub-model is obtained by training the training device 700 of the neural network model for image processing, which are used to predict the region of interest in the image and the threshold for binarizing the image, respectively. The neural network model can dynamically adjust and accurately determine the region of interest according to the actual image. The neural network model makes the boundaries of objects in the binary image clearer by dynamically adjusting the binarization threshold. Especially in complex scenes, such as stacked objects, large changes in lighting or complex backgrounds, the model can automatically generate the optimal threshold for each specific image. This dynamic adjustment avoids the errors that may be caused by the fixed threshold method, thereby ensuring that the binary image can accurately reflect the boundary between the object and the background, making the binarization result of the image clearer and more accurate.

[0123] In specific applications, the prediction thresholds generated by the model can significantly improve the quality of the binary image, which directly affects the accuracy of object counting. The clear object outlines enable subsequent recognition and counting methods to more accurately identify the boundaries of the objects, avoiding miscounting due to blurred or incomplete boundaries. In this way, the model ensures that the outlines of objects can be clearly segmented in various scenarios, thereby improving the overall accuracy of counting.

[0124] In addition, during the training process, the second sub-model can learn the binarization thresholds of various sample images to provide stable output under different types of images and object arrangements. This adaptive binarization processing method significantly improves the generalization ability of the model, enabling it to output high-quality binarization results when facing different image conditions, ensuring the accuracy of counting.

[0125] According to some embodiments, the first sub-model comprises a feature extraction network for extracting a feature map of an input image.

[0126] According to some embodiments, the sixth acquisition module includes: a fifth acquisition unit, configured to obtain the feature map extracted by the feature extraction network of the first sub-model for the second sample image; a sixth acquisition unit, configured to input the second output and the feature map into the second sub-model, and obtain the prediction threshold output by the second sub-model for binarizing the second sample image.

[0127] Therefore, the feature map is extracted by the feature extraction network of the first sub-model, and the second sub-model is trained in combination with the predicted coordinate values ​​of the region of interest output by the first sub-model.

[0128] According to some embodiments, the feature map extracted by the feature extraction network of the first sub-model for the input image includes a plurality of feature maps with different scales.

[0129] Therefore, the use of feature maps of different scales can capture different levels of information in the image. Feature maps of larger scales retain more spatial detail information and are suitable for detecting larger objects or structures in a larger range; while feature maps of smaller scales focus on more abstract, global features, which help to identify the overall outline and background of objects. This combination of multi-scale feature maps enables the model to consider both local details and global context at the same time, improving the model's perception ability. In addition, by combining multi-scale feature maps, the model can better cope with objects of various scales and different image resolutions. Regardless of whether the object is large or small, near or far in the image, the combination of multi-scale feature maps can ensure the stability of the model in different scenarios and reduce recognition and counting errors caused by changes in object size. At the same time, it enables the model to effectively cope with complex scenes and improve the generalization ability of processing diverse images.

[0130] According to some embodiments, during the training of the second sub-model, the parameters of the first sub-model remain fixed.

[0131] According to some embodiments, the second sub-model includes a scaling layer, a feature fusion layer, a pooling layer and a fully connected layer, and wherein the sixth acquisition module includes: a first acquisition unit, configured to input the multiple feature maps with different scales into the scaling layer, and obtain the feature map with the same scale output by the scaling layer; a second acquisition unit, configured to input the feature map with the same scale into the feature fusion layer to perform channel stitching on the feature maps with the same scale, and obtain the fused feature map output by the feature fusion layer; a third acquisition unit, configured to input the fused feature map and the second output into the pooling layer to map the predicted coordinate value of the minimum rectangular area in the second sample image that can contain the items to be counted to the fused feature map, and obtain the pooled feature map output by the pooling layer; a fourth acquisition unit, configured to input the pooled feature map into the fully connected layer, and obtain the prediction threshold output by the fully connected layer.

[0132] The hierarchical structure design (scaling layer, feature fusion layer, pooling layer, and fully connected layer) can more effectively utilize the multi-scale features of the image. The first acquisition unit can eliminate the size differences between feature maps of different scales by inputting feature maps of different scales into the scaling layer and scaling them to the same scale, making the subsequent feature fusion operation more unified. This operation ensures that the features learned by the model on feature maps of different scales can be effectively compared and combined at the same scale, thereby improving the accuracy of threshold prediction. The third acquisition unit maps the predicted coordinate values ​​of the region of interest to the fused feature map through the pooling layer, and pools the region of interest into a feature map of a fixed size, thereby achieving accurate positioning of the objects to be counted.

[0133] Therefore, the combination of scaling, feature fusion, pooling and fully connected layers enables the second sub-model to process not only image features of a single scale, but also feature maps from multiple scales. This multi-scale fusion mechanism enables the model to maintain good generalization capabilities when facing various image resolutions and object sizes, improves the accuracy of threshold prediction, and ensures the robustness of binarization processing in complex image scenes. In turn, it can improve the accuracy of subsequent object counting based on the processed images.

[0134] According to another aspect of the present disclosure, a device for image recognition is provided. Figure 8As shown, the image recognition device 800 includes: a seventh acquisition module 801, configured to acquire a first image containing objects to be counted, wherein in the first image, the objects to be counted are stacked along a first direction; a second determination module 802, configured to determine a threshold for binarizing the first image using a neural network model; an image processing module 803, configured to binarize the first image based on the threshold to obtain a second image; a first sliding module 804, configured to set a first sliding window in the second image, and make the first sliding window slide M times along a second direction in the second image, wherein the length of the first sliding window in the first direction is not less than the length of the second image in the first direction, and wherein the second direction is perpendicular to the first direction, and wherein M is an integer greater than 1; a third determination module 805, configured to identify the number Cnt of the objects to be counted contained in the first sliding window in each sliding i , wherein i∈[1,M]; and a fourth determination module 806, configured to determine the number of objects to be counted in the first image based on the number i∈[1,M] corresponding to each sliding of the first sliding window, wherein the neural network model is trained according to a training method for a neural network model for image processing.

[0135] According to some embodiments, the seventh acquisition module includes: a shooting unit, configured to shoot the objects to be counted to obtain an original image; and a capture unit, configured to determine an area corresponding to the objects to be counted in the original image, and capture the area as the first image.

[0136] Therefore, by determining the specific area where the objects to be counted are located and capturing it as the second image, the interference of irrelevant background is reduced and the accuracy of subsequent image processing is improved.

[0137] Exemplarily, in the first image, the objects to be counted are stacked along a first direction. Exemplarily, the first direction is a vertical direction, and the second direction is a horizontal direction.

[0138] The second determination module 802 uses a neural network model to determine a threshold for binarizing the first image, and can adaptively determine the binarization threshold to obtain an optimal binarization effect, thereby reducing the error of manually setting the threshold. Thus, the flexibility and adaptability of the image binarization process are guaranteed, so that the binarization result can better adapt to different image scenes, and improve the accuracy of subsequent object counting.

[0139] The image processing module 803 separates the objects to be counted from the background in the image through binarization processing, making the objects in the image more prominent, improving the effectiveness of image analysis, and facilitating subsequent object counting.

[0140] The first sliding module 804 sets a sliding window and slides in the image. The sliding window slides and counts in a direction perpendicular to the direction in which the objects are placed, so that the third determination module 805 and the fourth determination module 806 can count the number of objects at different positions to prevent the adhesion of objects at some positions from affecting the accuracy of counting, effectively reducing local counting errors. Through multiple sliding cumulative statistics, the comprehensiveness and accuracy of the counting are ensured.

[0141] According to some embodiments, the region is the smallest rectangular region in the first image that can contain the objects to be counted, thereby reducing interference from redundant background in the image and enhancing the accuracy of object counting based on the image.

[0142] According to some embodiments, the interception unit is further configured to: input the original image into the neural network model, and obtain the coordinate value of the area output by the neural network model. Thus, the interception unit directly obtains the coordinate value of the area of ​​interest in the image using the neural network model, which can improve the accuracy of area positioning, avoid manual marking errors, and further improve the automation of the counting scheme.

[0143] Therefore, by using a neural network model to determine the region of interest and the threshold for binarizing the image, the region of interest and the threshold can be dynamically adjusted and determined according to the actual image. Binarizing the image based on the threshold obtained by dynamic prediction can improve the accuracy of object counting when subsequently counting objects in the above-mentioned processed image.

[0144] According to some embodiments, the first sliding module 804 is further configured to: enable the first sliding window to slide M times along the second direction in the second image with a first step length, so as to slide from a first edge of the second image to a second edge of the second image.

[0145] The first sliding module 804 sets a first sliding window in the second image and slides it from the first edge of the second image to the second edge of the second image along the second direction, ensuring that all features of the objects to be counted are captured at different positions. By ensuring that the length of the sliding window in the first direction is not less than the length of the image, the entire image can be covered in the first direction, thereby ensuring that no objects to be counted are missed. This setting helps to improve the comprehensiveness and accuracy of counting and avoid errors caused by uneven distribution of objects in the image.

[0146] According to some embodiments, the image recognition device 800 also includes: a setting module, configured to set a second sliding window in the second image, wherein the length of the second sliding window in the first direction is not less than the length of the first sliding window in the first direction; a second sliding module, configured to slide the second sliding window N times in the first sliding window along the second direction with a second step length before sliding the first sliding window, wherein N is an integer greater than 1; and a third sliding module, configured to slide the second sliding window N times in the first sliding window along the second direction with a second step length after each of the M slidings of the first sliding window.

[0147] According to some embodiments, the second sliding module and the third sliding module are further configured to: cause the second sliding window to slide N times in the first sliding window along the second direction with the second step length, so as to slide from a first edge of the first sliding window to a second edge of the first sliding window.

[0148] Therefore, by setting a second sliding window within the first sliding window for refined sliding, the precision of counting is further improved. The operation of the second sliding window helps to capture smaller local changes and details, avoiding missing smaller objects or counting errors in complex scenes. The combined use of this multi-layer sliding window can significantly improve the accuracy and robustness of object counting in the image.

[0149] According to some embodiments, the third determination module includes: a first determination unit configured to determine the number Cnt of the objects to be counted contained in the second sliding window in each sliding of the second sliding window during the process in which the second sliding window slides from the first edge of the first sliding window to the second edge of the first sliding window. j , wherein j∈[1,N]; and a second determining unit, configured to determine the number Cnt corresponding to each sliding of the second sliding window j , determine the number Cnt of the objects to be counted contained in the first sliding window i .

[0150] According to some embodiments, the second determining unit is further configured to: in the process of the second sliding window sliding from the first edge of the first sliding window to the second edge of the first sliding window, based on the number Cnt corresponding to each sliding of the second sliding window j The mode of the first sliding window is used to determine the number of the objects to be counted contained in the first sliding window. i .

[0151] During the sliding process of the second sliding window, the number of local objects is determined based on the mode of each sliding, which can effectively reduce the false counting caused by noise or irregular shapes in the image. As a statistical method, the mode can provide a stable counting method to avoid the influence of extreme values ​​on the results, thereby improving the stability and reliability of counting.

[0152] According to some embodiments, the fourth determining module is further configured to: in the process of the first sliding window sliding from the first edge of the second image to the second edge of the second image, based on the number Cnt corresponding to each sliding of the first sliding window i The maximum value among them is used to determine the number of the objects to be counted in the first image.

[0153] During the sliding process of the first sliding window, the number of objects in the overall image is determined based on the maximum value of each sliding, which can ensure that the final result fully takes into account the objects in possible overlapping or dense areas, thereby minimizing omissions and improving the accuracy of the counting results.

[0154] Therefore, the image recognition device 800 further refines the counting process by introducing the sliding window counting method, making the counting result more accurate, while reducing possible omissions in the counting process, and improving the stability and accuracy of object counting.

[0155] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the aforementioned methods.

[0156] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute any of the aforementioned methods.

[0157] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements any of the aforementioned methods when executed by a processor.

[0158] like Fig. 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0159] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the electronic device 900. The input unit 906 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 908 can include but is not limited to a disk and an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TM devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0160] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a training method for a neural network model for image processing and a method for image recognition. For example, in some embodiments, the training method for a neural network model for image processing and a method for image recognition may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method for a neural network model for image processing and the method for image recognition described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other appropriate manner (eg, by means of firmware) to execute a training method for a neural network model for image processing and a method for image recognition.

[0161] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0163] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0165] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0166] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0168] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A training method for a neural network model for image processing, wherein: The neural network model includes a first sub-model and a second sub-model, and wherein the training method includes a training method for the first sub-model and a training method for the second sub-model, and the training method for the first sub-model includes: Acquire a first sample image containing a first object to be counted, and mark the real coordinate values ​​of the region of interest in the first sample image; Inputting the first sample image into the first sub-model, obtaining a first output of the first sub-model, wherein the first output represents predicted coordinate values ​​of a region of interest in the first sample image; Based on the real coordinate value and the predicted coordinate value, adjusting the parameters of the first sub-model, The training method of the second sub-model includes: In response to completion of the training of the first sub-model, a second sample image containing second objects to be counted is acquired, and a true threshold for binarizing the second sample image is marked, wherein the contour of the second objects to be counted in the second sample image binarized based on the true threshold meets a clarity standard; Inputting the second sample image into the first sub-model to obtain a second output of the first sub-model, wherein the second output represents predicted coordinate values ​​of the region of interest of the second sample image; Inputting the second output into the second sub-model to obtain a prediction threshold output by the second sub-model for binarizing the second sample image; Calculating a loss value based on the true threshold and the predicted threshold; and adjusting the parameters of the second sub-model based on the loss value, The second sample image, after being binarized based on the prediction threshold, can be used to identify and count the second objects to be counted contained therein.

2. The method according to claim 1, wherein: The first sub-model includes a feature extraction network for extracting a feature map from the input image, and wherein the step of inputting the second output into the second sub-model to obtain a prediction threshold output by the second sub-model for binarizing the second sample image includes: Obtaining a feature map extracted by the feature extraction network of the first sub-model for the second sample image; The second output and the feature map are input into the second sub-model to obtain the prediction threshold output by the second sub-model for binarizing the second sample image.

3. The method according to claim 2, wherein: The feature map extracted by the feature extraction network of the first sub-model for the input image includes multiple feature maps with different scales.

4. The method according to claim 3, wherein: The second sub-model includes a scaling layer, a feature fusion layer, a pooling layer, and a fully connected layer, and wherein the step of inputting the second output and the feature map into the second sub-model to obtain a prediction threshold output by the second sub-model for binarizing the second sample image includes: Inputting the plurality of feature maps with different scales into the scaling layer, and obtaining feature maps with the same scale output by the scaling layer; Inputting the feature maps with the same scale into the feature fusion layer to perform channel splicing on the feature maps with the same scale, and obtaining a fused feature map output by the feature fusion layer; Inputting the fused feature map and the second output into the pooling layer to map the predicted coordinate values ​​of the minimum rectangular area that can contain the objects to be counted in the second sample image to the fused feature map, and obtaining the pooled feature map output by the pooling layer; The pooled feature map is input into the fully connected layer, and the prediction threshold output by the fully connected layer is obtained.

5. The method according to any one of claims 1 to 4, wherein: The region of interest is the smallest rectangular area in the image that can contain the objects to be counted in the image.

6. The method according to any one of claims 1 to 5, wherein: During the training of the second sub-model, the parameters of the first sub-model remain fixed.

7. A method for image recognition, comprising: Acquire a first image containing objects to be counted, wherein in the first image, the objects to be counted are stacked along a first direction; Determining a threshold for binarizing the first image using a neural network model; Based on the threshold, binarize the first image to obtain a second image; Setting a first sliding window in the second image, and making the first sliding window slide M times along a second direction in the second image, wherein a length of the first sliding window in the first direction is not less than a length of the second image in the first direction, and wherein the second direction is perpendicular to the first direction, and wherein M is an integer greater than 1; Identify the number Cnt of the objects to be counted contained in the first sliding window in each sliding i , where i∈[1,M]; and Based on the number Cnt corresponding to each sliding of the first sliding window i , determining the number of the objects to be counted in the first image, Wherein, the neural network model is trained according to the method described in any one of claims 1-6.

8. The method according to claim 7, further comprising: Setting a second sliding window in the second image, wherein a length of the second sliding window in the first direction is not less than a length of the first sliding window in the first direction; Before causing the first sliding window to start sliding, causing the second sliding window to slide N times in the first sliding window along the second direction with a second step length, where N is an integer greater than 1; and After each of the M slidings of the first sliding window, the second sliding window is caused to slide N times in the first sliding window along the second direction with the second step length.

9. The method according to claim 8, wherein: The making the second sliding window slide N times in the first sliding window along the second direction with a second step length comprises: The second sliding window is caused to slide N times in the first sliding window along the second direction with the second step length, so as to slide from the first edge of the first sliding window to the second edge of the first sliding window, and wherein, The making the first sliding window slide M times along the second direction in the second image comprises: The first sliding window is caused to slide M times along the second direction in the second image with a first step length, so as to slide from a first edge of the second image to a second edge of the second image.

10. The method according to claim 9, wherein: The identification of the number of the objects to be counted contained in the first sliding window in each sliding i include: In the process that the second sliding window slides from the first edge of the first sliding window to the second edge of the first sliding window, Determine the number Cnt of the objects to be counted contained in the second sliding window in each sliding of the second sliding window j , where j∈[1,N]; and Based on the number Cnt corresponding to each sliding of the second sliding window j , determine the number Cnt of the objects to be counted contained in the first sliding window i .

11. The method according to claim 10, wherein the number Cnt corresponding to each sliding of the first sliding window is i , determining the number of the objects to be counted in the first image includes: In the process of the first sliding window sliding from the first edge of the second image to the second edge of the second image, based on the number Cnt corresponding to each sliding of the first sliding window i The maximum value among them is used to determine the number of the objects to be counted in the first image.

12. The method according to claim 10 or 11, wherein: The number Cnt corresponding to each sliding of the second sliding window j , determine the number Cnt of the objects to be counted contained in the first sliding window i include: In the process of the second sliding window sliding from the first edge of the first sliding window to the second edge of the first sliding window, based on the number Cnt corresponding to each sliding of the second sliding window j The mode of the first sliding window is used to determine the number of the objects to be counted contained in the first sliding window. i .

13. The method according to any one of claims 7 to 12, wherein: The acquiring of a first image containing objects to be counted comprises: Photographing the objects to be counted to obtain an original image; Determine the area corresponding to the objects to be counted in the original image, and capture the area as the first image.

14. The method according to claim 13, wherein: The determining the area corresponding to the objects to be counted in the original image comprises: The original image is input into the neural network model to obtain the coordinate value of the area output by the neural network model.

15. The method according to claim 13 or 14, wherein: The region is the smallest rectangular region in the original image that can contain the objects to be counted.

16. A training device for a neural network model for image processing, wherein: The neural network model includes a first sub-model and a second sub-model, and wherein the training device includes a training device for the first sub-model and a training device for the second sub-model, and the training device for the first sub-model includes: A first acquisition module is configured to acquire a first sample image containing a first object to be counted, and mark the real coordinate value of the region of interest in the first sample image; A second acquisition module is configured to input the first sample image into the first sub-model to acquire a first output of the first sub-model, where the first output represents a predicted coordinate value of a region of interest in the first sample image; a first adjustment module, configured to adjust parameters of the first sub-model based on the real coordinate value and the predicted coordinate value, in, The training device of the second sub-model comprises: a third acquisition module, configured to, in response to completion of the training of the first sub-model, acquire a second sample image containing second objects to be counted, and mark a true threshold for binarizing the second sample image, wherein the contour of the second objects to be counted in the second sample image binarized based on the true threshold meets a clarity standard; a fourth acquisition module, configured to input the second sample image into the first sub-model, and acquire a second output of the first sub-model, wherein the second output represents a predicted coordinate value of a region of interest of the second sample image; a sixth acquisition module, configured to input the second output into the second sub-model, and acquire a prediction threshold output by the second sub-model for binarizing the second sample image; A calculation module, configured to calculate a loss value based on the real threshold and the predicted threshold; and a second adjustment module, configured to adjust parameters of the second sub-model based on the loss value, The second sample image, after being binarized based on the prediction threshold, can be used to identify and count the second objects to be counted contained therein.

17. The device according to claim 16, wherein: The first sub-model comprises a feature extraction network for extracting a feature map from an input image, and wherein the sixth acquisition module comprises: a fifth acquisition unit, configured to acquire a feature map extracted by the feature extraction network of the first sub-model for the second sample image; The sixth acquisition unit is configured to input the second output and the feature map into the second sub-model, and acquire the prediction threshold output by the second sub-model for binarizing the second sample image.

18. The device according to claim 17, wherein: The feature map extracted by the feature extraction network of the first sub-model for the input image includes multiple feature maps with different scales.

19. The device according to claim 18, wherein: The second sub-model includes a scaling layer, a feature fusion layer, a pooling layer and a fully connected layer, and wherein the sixth acquisition module includes: A first acquisition unit is configured to input the plurality of feature maps with different scales into the scaling layer, and acquire a feature map with the same scale output by the scaling layer; A second acquisition unit is configured to input the feature maps with the same scale into the feature fusion layer to perform channel splicing on the feature maps with the same scale, and acquire a fused feature map output by the feature fusion layer; a third acquisition unit, configured to input the fused feature map and the second output into the pooling layer, so as to map the predicted coordinate value of the minimum rectangular area that can contain the objects to be counted in the second sample image to the fused feature map, and acquire the pooled feature map output by the pooling layer; The fourth acquisition unit is configured to input the pooled feature map into the fully connected layer and obtain the prediction threshold output by the fully connected layer.

20. The device according to any one of claims 16 to 19, wherein: The region of interest is the smallest rectangular area in the image that can contain the objects to be counted.

21. The device according to any one of claims 16 to 20, wherein: During the training of the second sub-model, the parameters of the first sub-model remain fixed.

22. An image recognition device, comprising: a seventh acquisition module, configured to acquire a first image containing objects to be counted, wherein in the first image, the objects to be counted are stacked along a first direction; A second determination module is configured to determine a threshold for binarizing the first image using a neural network model; an image processing module, configured to binarize the first image based on the threshold value to obtain a second image; a first sliding module, configured to set a first sliding window in the second image, and make the first sliding window slide M times along a second direction in the second image, wherein a length of the first sliding window in the first direction is not less than a length of the second image in the first direction, and wherein the second direction is perpendicular to the first direction, and wherein M is an integer greater than 1; The third determining module is configured to identify the number Cnt of the objects to be counted contained in the first sliding window in each sliding. i , where i∈[1,M]; and The fourth determining module is configured to determine the number Cnt corresponding to each sliding of the first sliding window based on the number Cnt corresponding to each sliding of the first sliding window i , determining the number of the objects to be counted in the first image, Wherein, the neural network model is trained according to the method described in any one of claims 1-6.

23. The apparatus according to claim 22, further comprising: a setting module configured to set a second sliding window in the second image, wherein a length of the second sliding window in the first direction is not less than a length of the first sliding window in the first direction; a second sliding module configured to, before sliding the first sliding window, make the second sliding window slide N times in the first sliding window along the second direction with a second step length, wherein N is an integer greater than 1; and The third sliding module is configured to make the second sliding window slide N times in the first sliding window along the second direction with a second step length after each sliding of the M sliding of the first sliding window.

24. The device according to claim 23, wherein: The second sliding module and the third sliding module are further configured as: The second sliding window is caused to slide N times in the first sliding window along the second direction with the second step length, so as to slide from the first edge of the first sliding window to the second edge of the first sliding window, wherein: The first sliding module is further configured as: The first sliding window is caused to slide M times along the second direction in the second image with a first step length, so as to slide from a first edge of the second image to a second edge of the second image.

25. The device according to claim 24, wherein: The third determination module comprises: a first determining unit configured to determine, during a process in which the second sliding window slides from a first edge of the first sliding window to a second edge of the first sliding window, a number Cnt of the objects to be counted contained in the second sliding window in each sliding of the second sliding window; j , where j∈[1,N]; and The second determining unit is configured to determine the number Cnt corresponding to each sliding of the second sliding window based on the number Cnt corresponding to each sliding of the second sliding window j , determine the number Cnt of the objects to be counted contained in the first sliding window i .

26. The device according to claim 25, wherein The fourth determining module is further configured to: In the process of the first sliding window sliding from the first edge of the second image to the second edge of the second image, based on the number Cnt corresponding to each sliding of the first sliding window i The maximum value among them is used to determine the number of the objects to be counted in the first image.

27. The device according to claim 25 or 26, wherein: The second determining unit is further configured to: In the process of the second sliding window sliding from the first edge of the first sliding window to the second edge of the first sliding window, based on the number Cnt corresponding to each sliding of the second sliding window j The mode of the first sliding window is used to determine the number of the objects to be counted contained in the first sliding window. i .

28. The device according to any one of claims 22 to 27, wherein: The seventh acquisition module comprises: A photographing unit, configured to photograph the objects to be counted to obtain an original image; The interception unit is configured to determine a region corresponding to the objects to be counted in the original image, and intercept the region as the first image.

29. The device according to claim 28, wherein The interception unit is further configured to: The first original is input into the neural network model to obtain the coordinate value of the area output by the neural network model.

30. The device according to claim 28 or 29, wherein: The region is the smallest rectangular region in the original image that can contain the objects to be counted.

31. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

33. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images