A catering tax source monitoring target tracking method and system

Abstract

The present application discloses a catering tax source monitoring target tracking algorithm and system. The method may include: obtaining the area including the target and its surroundings and the pre-search area in the current frame image, wherein, for the first frame image, manually initialize the area frame and pre-search area of ​​the target to be tracked and its surroundings; input the target and its surrounding area and the target pre-search area into the pre-trained convolutional neural network respectively, extract the convolution features of the fourth layer and the fifth layer respectively, and then input them into the fully connected layer, and obtain the output of the fully connected layer; use the correlation filter to calculate the output responses obtained by the two fourth layers and the fifth layer; weightedly add the above two response results to obtain the response of the semantic branch; input the target search areas corresponding to the three responses with the highest scores into the IoU prediction network respectively to obtain the output results of the network; compare the three output results, and the target area corresponding to the maximum value is the tracking result of the current frame.

Description

Technical Field

[0001] The present application relates to the field of CV (Computer Vision), and more specifically, to a catering tax source monitoring target tracking method and system. Background Art

[0002] Visual object tracking is an important research direction in computer vision and has a wide range of applications, such as video surveillance, human-computer interaction, and unmanned driving. In the past two or three decades, visual object tracking technology has made great progress. In particular, in the past two years, the object tracking method using deep learning has achieved satisfactory results, making object tracking technology a breakthrough.

[0003] Visual tracking refers to the detection, extraction, identification and tracking of moving targets in image sequences to obtain the motion parameters of the moving targets, such as position, speed, acceleration and motion trajectory, so as to carry out the next step of processing and analysis, realize the understanding of the behavior of the moving targets and complete a higher level of detection tasks.

[0004] The prior art provides a tracking algorithm based on region, a tracking method based on feature, a tracking method based on contour, a tracking method based on model, and a tracking algorithm based on detection. Recently, with the rapid development of artificial intelligence, some people have proposed to apply neural network technology to tracking algorithms.

[0005] In response to the urgent needs of my country's catering and hotel taxation practice, a catering tax source monitoring target tracking algorithm and system was developed. The system comprehensively utilizes innovative technological achievements in multiple fields such as artificial intelligence, big data, and the Internet, and realizes concurrent video stream acquisition and storage management, real-time video data target tracking, and provides a basis for subsequent hotel tax statistics based on video target behavior and per capita hotel consumption network evaluation. It has the obvious advantages of strong operability and easy promotion and application. Summary of the invention

[0006] The present application provides a catering tax source monitoring target tracking method and system. By combining a deep learning network with an IoU (Intersection over Union) network, accurate tracking of deformable targets is achieved.

[0007] One aspect of the present application provides a catering tax source monitoring target tracking method, comprising:

[0008] 1) obtaining an area including the target and its surroundings and a pre-search area in the current frame image, wherein for the first frame image, manually initializing an area frame of the target to be tracked and its surroundings and a pre-search area;

[0009] 2) Inputting the target and its surrounding area into a pre-trained convolutional neural network, extracting the convolutional features of the fourth layer, and then inputting them into the fully connected layer to obtain the output of the fully connected layer;

[0010] 3) Inputting the target and its surrounding area into the pre-trained convolutional neural network, extracting the convolutional features of the fifth layer, and then inputting them into the fully connected layer to obtain the output of the fully connected layer;

[0011] 4) inputting the target pre-search area into the pre-trained convolutional neural network, extracting the convolutional features of the fourth layer, and then inputting it into the fully connected layer, and obtaining the output of the fully connected layer;

[0012] 5) inputting the target pre-search area into the pre-trained convolutional neural network, extracting the convolutional features of the fifth layer, and then inputting it into the fully connected layer, and obtaining the output of the fully connected layer;

[0013] 6) Use the correlation filter to calculate the response of the fully connected layer output of step 2) and step 4);

[0014] 7) Use the correlation filter to calculate the response of the fully connected layer output of step 3) and step 5);

[0015] 8) weighted addition of the results of step 6) and step 7) to obtain the response of the semantic branch;

[0016] 9) The target boxes corresponding to the three responses with the highest scores are taken as candidate results and input into the IoU prediction network respectively to obtain the output results of the network;

[0017] 10) Compare the three output results, and the target area corresponding to the maximum value is the tracking result of the current frame;

[0018] 11) Perform the next frame image operation and go to step 1) until all tracking is completed.

[0019] Another aspect of the present application provides a catering tax source monitoring target tracking system, comprising:

[0020] A memory storing computer-executable instructions; and a processor executing the instructions:

[0021] 1) obtaining an area including the target and its surroundings and a pre-search area in the current frame image, wherein for the first frame image, manually initializing an area frame of the target to be tracked and its surroundings and a pre-search area;

[0022] 2) Inputting the target and its surrounding area into a pre-trained convolutional neural network, extracting the convolutional features of the fourth layer, and then inputting them into the fully connected layer to obtain the output of the fully connected layer;

[0023] 3) Inputting the target and its surrounding area into the pre-trained convolutional neural network, extracting the convolutional features of the fifth layer, and then inputting them into the fully connected layer to obtain the output of the fully connected layer;

[0024] 4) inputting the target pre-search area into the pre-trained convolutional neural network, extracting the convolutional features of the fourth layer, and then inputting it into the fully connected layer, and obtaining the output of the fully connected layer;

[0025] 5) inputting the target pre-search area into the pre-trained convolutional neural network, extracting the convolutional features of the fifth layer, and then inputting it into the fully connected layer, and obtaining the output of the fully connected layer;

[0026] 6) Use the correlation filter to calculate the response of the fully connected layer output of step 2) and step 4);

[0027] 7) Use the correlation filter to calculate the response of the fully connected layer output of step 3) and step 5);

[0028] 8) weighted addition of the results of step 6) and step 7) to obtain the response of the semantic branch;

[0029] 9) Input the target search areas corresponding to the three responses with the highest scores into the IoU prediction network respectively to obtain the output results of the network;

[0030] 10) Compare the three output results, and the target area corresponding to the maximum value is the tracking result of the current frame;

[0031] 11) Perform the next frame image operation and go to step 1) until all tracking is completed.

[0032] In one embodiment, the step 9) comprises:

[0033] 9-1) Input the real area and border information of the target in the previous frame to build the reference branch of the network;

[0034] 9-2) Input the target search area information corresponding to the three responses with the highest scores and construct a test branch.

[0035] In one embodiment, the step 9-1) comprises:

[0036] Input the real target area to the ResNet-18 network, extract the first-layer feature map, obtain the feature map after convolution and pooling, perform full convolution operation, and input the result to the test branch;

[0037] The second layer feature map is extracted, and after convolution and pooling, the feature map is obtained, and a full convolution operation is performed, and the result is input to the test branch.

[0038] In one embodiment, the step 9-2) comprises:

[0039] Input the real target area into the ResNet-18 network, extract the first layer feature map, and obtain the feature map after convolution and pooling; extract the second layer feature map, and obtain the feature map after convolution and pooling; combine the results of the reference branch to obtain the IoU value.

[0040] Beneficial Effects

[0041] (1) The highest convolutional feature is used as the input of the semantic branch, which is robust to changes in the appearance of the target;

[0042] (2) Different from the traditional classification-based tracking method, the proposed method uses the IoU network to refine the bounding box, which contains more information about the spatial state of the target;

[0043] (3) The hybrid strategy provides complementary benefits. The semantic branch obtains the rough localization of the target, and the Iou network branch refines the bounding box;

[0044] (4) Offline training of the two branches improves tracking speed;

[0045] (5) It can count the number of customers in real time and provide a basis for taxation of the catering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the purpose, system solution and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings:

[0047] Figure 1 The figure shows the network structure of ResNet-18 according to one embodiment of the present application.

[0048] Figure 2 Shown is a flow chart of a catering tax source monitoring target tracking algorithm according to one embodiment of the present application.

[0049] Figure 3 Shown Figure 2 Schematic block diagram of the process shown.

[0050] Figure 4 Shown is a schematic diagram of the overlap (IoU) prediction network structure.

[0051] Figure 5 Shown is a schematic diagram of overlap calculation according to one embodiment of the present application.

[0052] Figure 6 The figure shows a block diagram of an embodiment of the present application when implemented using a computer architecture.

[0053] Figure 7 Shown is a diagram of tracking results according to one embodiment of the present application. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and implementation examples. The specific embodiments described in this application are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings.

[0055] Reference in this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present disclosure. The appearances of the phrase "an embodiment" in various places in this specification are not necessarily all referring to the same embodiment.

[0056] In this application, the accuracy and operation efficiency of the system are comprehensively considered, and the classic ResNet-18 network structure is used to implement various implementation methods of this application. Figure 1 The following is an exemplary ResNet-18 network structure. Figure 1 As shown, the network structure includes 17 convolutional layers (201) and 1 fully connected layer (203).

[0057] Specifically, Figure 1 As shown, first, a convolution layer is constructed using a 7×7 filter with a stride of 2. Assuming that the network input size is m×n×3, in order to ensure that the first two dimensions of the feature matrix after convolution are the same as the first two dimensions of the input matrix, that is, m×n. That is, three circles of 0 are added to the outside of the input matrix. Then a maximum pooling layer (202) is constructed using a 3×3 filter with a stride of 2. The first and second basic blocks will obtain the feature matrix through two 3×3 convolutions with a stride of 1, each time adding a circle of 0. In this way, the first two dimensions of the feature matrix after convolution are still: m×n. The third basic block adds a 1×1 filter with a stride of 2 to construct a residual block for dimensionality increase, and adds the results of a 3×3 convolution with a stride of 2 and a 3×3 convolution with a stride of 1 to the output. The fourth basic block will obtain the result of the third basic block through two 3×3 convolutions with a stride of 1. Repeat the structure of basic block 3 and basic block 4 twice. All activation functions used previously use relu. After several rounds of such operations, the final 7×7×512 feature map is put into a 3×3 filter with a stride of 2 to construct an average pooling layer (202), and then softmax activation is performed (204). Although a specific ResNet-18 network structure is given here, those skilled in the art should understand that other network architectures can also be used without departing from the teachings of this application.

[0058] After re-constructing the above network, it is trained by using the ImageNet data set. The ImageNet data set is divided into a training set and a test set. The data set corresponds to, for example, 1000 categories. Each data has a corresponding label vector, and each label vector corresponds to a different category. This application does not care about the specific classification of the input image, but only uses the data set to train the weight of the ResNet-18 network. Specifically, the above ImageNet training set is adjusted to a size of 224×224×3, and then input into the ResNet-18 network to train the network to obtain the weight parameter information of each layer or unit of the network. Then, a predetermined test data set and a label vector of the corresponding category are input into the trained ResNet-18 network structure. The size of the test data set can be, for example, also 224×224×3. By inputting the above test data set and the label vector of the corresponding category into the ResNet-18 network, the output result of the ResNet-18 network can be detected, and the detected result is compared with the standard data to adjust the parameters (weights) of the ResNet-18 network according to the comparison error. Repeat the above steps until the test accuracy reaches a predetermined standard, for example, an accuracy rate of more than 98%.

[0059] like Figure 2 Shown is a flow chart of a catering tax source monitoring target tracking algorithm according to one embodiment of the present application. Figure 3 Shown Figure 2 Schematic block diagram of the process shown. Figure 4 The following is a schematic diagram of the overlap (IoU) prediction network structure. Figure 2 , Figure 3 as well as Figure 4 The embodiments disclosed in this application are described in detail.

[0060] like Figure 2As shown, according to one embodiment of the present application, a catering tax source monitoring target tracking algorithm includes: step S101, obtaining the current frame target and its surrounding area, pre-search area, and manually initializing the first frame image to be tracked target and its surrounding area frame and pre-search area; step S102, inputting the target and its surrounding area to CNN, extracting the convolution features of the fourth layer, and inputting them to the fully connected layer; step S103, inputting the target and its surrounding area to CNN, extracting the convolution features of the fifth layer, and inputting them to the fully connected layer; step S104, inputting the target pre-search area to CNN, extracting the convolution features of the fourth layer, and inputting them to the fully connected layer ; Step S105, input the target pre-search area into CNN, extract the convolution features of the fifth layer, and input them into the fully connected layer; Step S106, calculate the responses of the fully connected layer outputs of S102 and S104; Step S107, calculate the responses of the fully connected layer outputs of S103 and S105; Step S108, weighted addition of the results of S106 and S107, to obtain the responses of the semantic branches; Step S109, respectively input the three target frames with the highest scores into the IoU prediction network; Step S110, the target area corresponding to the maximum value is used as the tracking result of the current frame; Step S111, read the next frame image and go to S101.

[0061] Figure 3 The network structure of the tracker is shown in Figure 1. The algorithm consists of a semantic branch and a position branch. The function of the position branch is mainly implemented by the IoU prediction network branch. Figure 3 The dotted box 300 is the semantic branch. 309 is the IoU prediction network branch, and the specific structure is as follows Figure 4 The output result of the semantic branch is input into the network branch to obtain the tracking result 318.

[0062] The low-level features based on CNN contain more spatial information, while the high-level features contain more semantic information. The semantic branch is based on the high-level features and related filters of the CNN network. The output of the semantic branch is part of the input of the position branch, and the position branch is implemented by the IoU prediction network.

[0063] The steps S101 to S111 will be described below respectively.

[0064] Step S101, as Figure 2 and Figure 3 As shown, if it is the first frame of video image, manually initialize the area frame and pre-search area of ​​the target to be tracked and its surroundings in the first frame image; otherwise, obtain the area and pre-search area of ​​the target and its surroundings in the current frame according to the tracking result of the previous frame.

[0065] like Figure 3As shown in FIG. 1 , first, the target and its surrounding area R numbered 301 and the search area set S numbered 310 are input into the semantic branch. Let T represent the precise target area, R represent the target and its surrounding area, and S represent the search area set. The sizes of R and S are both W. S ×H S ×3, the size of T is W T ×H T ×3.W S and H S represents the width and height of regions R and S, W T and H T Respectively represent the width and height of area T. Since the precise target area T is smaller than the target and its surrounding area R, the relationship between the above changes is W S >W T and H S >H T . Search set S = {s i}(i=1..n), where s i is the i-th candidate search area, n is the total number of candidate search areas. The size of the candidate search block is the same as the size of the target area T.

[0066] Step S102, input the target and its surrounding area into CNN, extract the convolution features of the fourth layer, and input them into the full connection layer; corresponding to number 303, calculate the fourth layer convolution of the target and its surrounding area R in the ResNet-18 network, obtain the convolution features of number 308, and then perform the full connection operation in number 305.

[0067] Step S103, input the target and its surrounding area to CNN, extract the convolution features of the fifth layer, and input them to the fully connected layer; in number 302, calculate the fifth layer convolution of the target and its surrounding area R in the ResNet-18 network to obtain the convolution features of number 304, and then number 306 performs a full connection operation.

[0068] Step S104, input the target pre-search area to CNN, extract the convolution features of the fourth layer, and input it to the fully connected layer; in number 319, calculate the convolution of a search area in the fourth layer of the ResNet-18 network, obtain the convolution features of number 313, and then perform a full connection operation in number 320.

[0069] Step S105, input the target pre-search area into CNN, extract the convolution features of the fifth layer, and input them into the fully connected layer; in number 311, calculate the convolution of a search area in the fifth layer of the ResNet-18 network, obtain the convolution features of number 312, and then perform the full connection operation in number 314.

[0070] Step S106 , calculate the response of the full link layer output of S102 and S104 ; obtain the full link response shown in 303 and 305 .

[0071] Step S107 , calculate the responses of the full-link layer outputs of S103 and S105 ; obtain the responses of the full-link layer shown in 306 and 314 .

[0072] Step S108, weighted addition of the results of S106 and S107, to obtain the response of the semantic branch; calculate the correlation coefficient as shown in the figure in number 307 and number 315; then number 316 performs weighted summation to obtain the three candidate regions with the highest correlation coefficient scores, numbered 317. This is also the output of the semantic branch of the algorithm.

[0073] In the image classification task, we used a CNN and kept all parameters unchanged. Input (R, S) to the CNN and extract the features of the last two convolutional layers as the semantic features of each image block. Let the two features of CNN be represented by f r1 (·) and f r2 (·) indicates that the response map of the correlation filter of the last two convolutional layers is

[0074] h1(R,S)=cor(f r1 (R),f r1 (S)), (1)

[0075] and

[0076] h2(R,S)=cor(f r2 (R),f r2 (S)), (2)

[0077] where cor(·) is a correlation operation. The response graph of the semantic branch is expressed as

[0078] h(R,S)=λh1(R,S)+(1-λ)h2(R,S), (3)

[0079] Among them, λ is a weighting parameter used to balance the influence of the two additive terms on the result. The training method of all parameters is the same as that of similar learning problems. For each data pair (R i ,S i ), Y i Represents the true value feature map of the corresponding search area. Where i = 1..n, n is the total number of search areas.

[0080] The semantic branch is optimized by minimizing the logistic loss function Los(·), as follows:

[0081]

[0082] where wi is the network parameter, and n is the number of training samples.

[0083] like Figure 3 As shown, the specific process of the algorithm is:

[0084] Step 1: Input R into the ResNet-18 network and extract the feature map f of the fourth convolutional layer r1 (R). After the fully connected layer, the feature map is obtained. Extract the feature map f of the fifth convolutional layer r1 (R). After the fully connected layer, the feature map f is obtained r2 (R);

[0085] Step 2: The same process as step 1 is completed on S. We get the feature map f r1 (S) and f r2 (S);

[0086] Step 3: h1(R,S) calculates the response graph using formula (1);

[0087] Step 4: Calculate the response graph using formula (2) h2(R,S);

[0088] Step 5: Use formula (3) to calculate the response map of the semantic branch h(R,S).

[0089] Step S109, input the three highest scoring target boxes to the IoU prediction network respectively; input the three candidate regions and the precise target region T to the position branch, and obtain the IoU value for each candidate region and T through the IoU network. After refining each pair of input bounding boxes, the output is the region with the maximum IoU value. It is worth noting that these two branches are trained separately. The specific execution process of the IoU prediction network branch is as follows: Figure 4 shown.

[0090] First, the reference area 401 is input into ResNet-18, and the first layer of convolution is calculated as shown in number 404 to obtain the first layer of convolution features shown in number 405, and then convolution (number 407) and pooling (number 408) are performed to obtain features 409, and then a full-link operation 413 is performed.

[0091] Input 405 into ResNet-18 to calculate the second layer of convolution (numbered 406), obtain the second layer of convolution features shown in number 410, and then perform convolution (numbered 411) and pooling (numbered 412) to obtain features 414, and then perform full link operation 415.

[0092] At the same time, the three target regions 402 obtained by the semantic branch are respectively input into ResNet-18, and the first layer of convolution is calculated as shown in number 403 to obtain the first layer of convolution features shown in number 418, and then convolution (numbered 417) and pooling (numbered 419) are performed to obtain feature 420, which is then ANDed with 413, and then a full link operation 422 is performed.

[0093] Input 416 into ResNet-18 to calculate the second layer of convolution (numbered 418), obtain the second layer of convolution features shown in number 421, and then perform pooling (numbered 423) to obtain feature 426, which is then ANDed with 415, and then a full-link operation 425 is performed.

[0094] The results obtained in 422 and 425 are input into the IoU prediction module 424, and the area corresponding to the maximum value is the target area to be tracked.

[0095] like Figure 4 As shown in the figure, for the reference region branch, the input is the true value region T and its bounding box B0, the first and second layer feature maps are extracted from the pre-trained ResNet-18, and then followed by a pooling layer and a fully connected layer. For the features of the first layer, it returns a positive coefficient adjustment vector a1(T, B0) of size 1×1×D, and for the features of the second layer, it returns a positive coefficient adjustment vector a2(T, B0) of size 1×1×D. D is a variable representing the size of the third dimension.

[0096] like Figure 4 As shown, for the search region branch, the input is the search region set S = {S i}i=1,2,3 and its bounding box set B={B i}i=1,2,3, for each element in S, extract the first and second layer feature maps from the pre-trained ResNet-18, and then use the bounding box estimate B as the pooling layer. Figure 2 As shown in Fig. 2, extracting general features for IoU prediction with the test branch is a more complex task, thus requiring more convolutional layers and higher pooling resolution. It generates vectors v1(S) of size K1×K1×D for layer 1 and layer 2, respectively. i ,B i )i=1,2,3 and a vector v2(S i ,B i )i=1,2,3, where K1 and K2 are the spatial sizes of the pooling layer. Then, the reference coefficient output vectors a1 and a2 are used to adjust the two result vectors through channel-wise multiplication. Bounding box B i The prediction IoU formula is expressed as

[0097] IoU(B i )=ηIoU1(B i )+(1-η)IoU2(B i ), (5)

[0098] IoU1(B i )=g(a1(T,B0)·v1(S i ,B i )), (6)

[0099] IoU2(B i )=g(a2(T,B0)·v1(S i ,B i )), (7)

[0100] Where g is an IoU prediction module consisting of three fully connected layers. η is a weighting parameter used to balance the impact of the two layers on the results.

[0101] When training, we minimize the prediction error of Equation (5), and when estimating the target region, we maximize Equation (5).

[0102] Tracking results B TRUTH Defined as

[0103]

[0104] The specific calculation method of IoU value is as follows: Figure 5 As shown. The calculation of the degree of overlap (IoU) is as follows. Let quadrilateral ABCD represent the target affine bounding box tracked in the previous frame. Quadrilateral DEFG represents the affine bounding box to be compared. The overlapping area of ​​the two is polygon AECGH. Let △ represent the area, followed by the name of the figure, which represents the area of ​​the corresponding figure. For example: △AECGH represents the area of ​​polygon AECGH. Then △AECGH=△AEC+△ACG+△AGH. The degree of overlap IoU between quadrilateral ABCD and quadrilateral DEFG is defined as That is, the ratio of the overlapping area of ​​the two to the total area of ​​the two types minus the overlapping area.

[0105] Step S110 , the target area corresponding to the maximum value is used as the tracking result of the current frame; as shown in 318 .

[0106] Step S111, read the next frame image, and go to S101. The border of the partial tracking result is as follows Figure 7 As shown in the rectangular frames indicated by arrows 701, 702, 703, and 704.

[0107] Figure 6 The following is a deformable target tracking system according to another aspect of the present application. Figure 6, which shows a schematic structural diagram of an electronic system 600 suitable for implementing an embodiment of the present disclosure. Figure 6 The electronic system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 6 As shown, the electronic system 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic system 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic system 600 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0110] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0111] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] The above-mentioned computer-readable medium may be included in the above-mentioned electronic system (also referred to as "deformable target tracking system" in this article); or it may exist independently without being assembled into the electronic system. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic system: 1) obtains the area including the target and its surroundings and the pre-search area in the current frame image, wherein, for the first frame image, the area box and the pre-search area of ​​the target to be tracked and its surroundings are manually initialized; 2) inputs the target and its surrounding area into a pre-trained convolutional neural network, extracts the convolution features of the fourth layer, and then inputs it into the fully connected layer, and obtains the output of the fully connected layer; 3) inputs the target and its surrounding area into a pre-trained convolutional neural network, extracts the convolution features of the fifth layer, and then inputs it into the fully connected layer, and obtains the output of the fully connected layer; 4) inputs the target pre-search area into a pre-trained convolutional neural network, extracts the convolution features of the fourth layer, Then input it into the fully connected layer and obtain the output of the fully connected layer; 5) input the target pre-search area into the pre-trained convolutional neural network, extract the convolution features of the fifth layer, and then input it into the fully connected layer, and obtain the output of the fully connected layer; 6) use the correlation filter to calculate the response of the fully connected layer output of step 2) and step 4); 7) use the correlation filter to calculate the response of the fully connected layer output of step 2) and step 4); 8) weighted addition of the results of step 6) and step 7) to obtain the response of the semantic branch; 9) input the target search areas corresponding to the three responses with the highest scores into the IoU prediction network respectively to obtain the output result of the network; 10) compare the three output results, and the target area corresponding to the maximum value is the tracking result of the current frame; 11) perform the next frame image operation and go to step 1) until all tracking is completed.

[0113] In one embodiment, the step 9) includes: 9-1) inputting the real area and border information of the target in the previous frame to construct a reference branch of the network; 9-2) inputting the target search area information corresponding to the three responses with the highest scores to construct a test branch;

[0114] In one embodiment, step 9-1) includes: inputting the real target area into the ResNet-18 network, extracting the first layer feature map, obtaining the feature map after convolution and pooling, performing a full convolution operation, and inputting the result into the test branch; extracting the second layer feature map, obtaining the feature map after convolution and pooling, performing a full convolution operation, and inputting the result into the test branch;

[0115] In one embodiment, step 9-2) includes: inputting the real target area into the ResNet-18 network, extracting the first layer feature map, and obtaining the feature map after convolution and pooling; extracting the second layer feature map, and obtaining the feature map after convolution and pooling; and combining the results of the reference branch to obtain the IoU value.

[0116] The various algorithms and details in the deformable target tracking method according to the first aspect of the present application are also applicable to the above-mentioned deformable target tracking system 600, so most of their descriptions are omitted for the purpose of brevity.

[0117] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. This application provides a catering tax source monitoring target tracking method, including: 1) obtaining an area R including the target and its surroundings and a pre-search area S in the current frame image, wherein for the first frame image, manually initializing an area frame of the target to be tracked and its surroundings and a pre-search area; 2) Input the target and its surrounding area R into the pre-trained convolutional neural network, extract the convolutional features of the fourth layer, and then input it into the fully connected layer to obtain the output of the fully connected layer fr1(R) ; 3) For the target and its surrounding area R, extract the fifth layer of convolutional features, then input them into the fully connected layer, and obtain the output of the fully connected layer fr2(R) ; 4) Input the target pre-search area S into the pre-trained convolutional neural network, extract the convolutional features of the fourth layer, and then input it into the fully connected layer to obtain the output of the fully connected layer fr1( S); 5) For the target pre-search area S, extract the convolutional features of the fifth layer, then input it to the fully connected layer, and obtain the output of the fully connected layer fr2(S) ; 6) Calculate the relevant filter response diagram according to formula (1): h1(R,S)=cor(f r1 (R),f r1 (S)), (1) where cor(·) is the correlation operation; 7) Calculate the relevant filter response diagram according to formula (2): h2(R,S)=cor(f r2 (R),f r2 (S)), (2) 8) Weighted addition of the results of step 6) and step 7) is performed to obtain the response of the semantic branch. The weighted formula (3) is as follows: h(R,S)=λh1(R,S)+(1-λ)h2(R,S), (3) Where λ is a weighting parameter used to balance the impact of the two additive terms on the result; 9) The target boxes corresponding to the three responses with the highest scores are taken as candidate results and input into the IoU prediction network respectively to obtain the output results of the network; 10) Compare the three output results, and the target area corresponding to the maximum value is the tracking result of the current frame; 11) Perform the next frame image operation and go to step 1) until all tracking is completed.

2. The catering tax source monitoring target tracking method according to claim 1, wherein the step 9) comprises: 9-1) Input the real area and border information of the target in the previous frame to build the reference branch of the network; 9-2) Input the target search area information corresponding to the three responses with the highest scores and construct a test branch.

3. The catering tax source monitoring target tracking method according to claim 2, wherein: The step 9-1) comprises: Input the real target area to the ResNet-18 network, extract the first-layer feature map, obtain the feature map after convolution and pooling, perform full convolution operation, and input the result to the test branch; The second layer feature map is extracted, and after convolution and pooling, the feature map is obtained, and a full convolution operation is performed, and the result is input to the test branch.

4. The catering tax source monitoring target tracking method according to claim 2, wherein the step 9-2) comprises: Input the real target area into the ResNet-18 network, extract the first layer feature map, and obtain the feature map after convolution and pooling; The second layer feature map is extracted, and after convolution and pooling, the feature map is obtained; the result of the combined reference branch is used to obtain the IoU value.

Images